Quantifiable Research Evaluation Portfolio Review

1. Cooperative State Research, Education, and Extension ServicePlanning and Accountability Experiences with Quantifiable Research Evaluation: Portfolio Review Expert Panel AEA, Toronto, Ontario, Canada October 28, 2005 Cheryl J. Oros, Director Henry M. Doan, Djimé D. Adoum, and Sharon Stout

2. CSREES Mission and Functions • Knowledge production and dissemination to solve national problems/ meet needs/ seize opportunities identified in the Strategic Plan • Agency works through state and university partners via grants

3. Evaluation/ManagementCycle Planning/Decision Making Evaluation Feedback • Identify needs, problems, solutions, refinements • Conceptualization of Program • Formulation of Evaluation Questions and Design • Feedback of Evaluation Findings to Managers • Refinement of Program • Continued Funding Implementation • Actualize the Program Plan • Collection of Evaluation Data • Analysis of Data

4. USDA-CSREES Strategic Goals Fully Aligned (5 Goals) 1. Enhance economic opportunities for agricultural producers 2. Support increased economic opportunities and improved quality of life in rural America 3. Enhance protection and safety of the nation’s agriculture and food supply 4. Improve the nation’s nutrition and health 5. Protect and enhance the nation’s natural resource base and environment

5. Cascading Alignment Mission . Goal Strategic Objective Portfolio Knowledge Area Code Program/project

6. Alignment Example Goal 3: Enhance Protection & Safety of Food Supply Strategic Objective 3.2 b: Animal Protection Animal Protection Portfolio • Knowledge Areas • (KA 311) Improve the management of animal diseases • (KA 312) Increase knowledge & improve management of pests & external parasites • (KA 313) Increase knowledge to control internal parasites • (KA 314) Reduce losses in livestock, poultry, and fish • (KA 315) Develop and implement effective animal care

7. OMB PART • OMB’s Program Assessment Rating Tool • Program Purpose & Design • Strategic Planning • Program Management • Program Results • CSREES Goal 1 Portfolio reviewed 2004; Goals 3 & 5 in 2005; Goals 2 & 4 in 2006

8. Evaluating Research & Education Efforts: Portfolio Approach • OMB PART/BPI led to development of new portfolio assessment tool and measures • Portfolio analysis (meta-analysis) used to assess progress toward goals; guide announcements for grants • Uses OMB R&D criteria (relevance, quality, performance)

9. Portfolio Review Expert Panel (PREP) Process • Focus on outcomes rather than processes • Level of analysis is a portfolio identified via Knowledge Area codes in databases • A PREP scores portfolio progress & provides recommendations for Agency

10. Evidentiary materials • Tracks papers, citations, patents, products, educational efforts, adoption of products/ practices • Entire evaluation studies and special analyses • Budget tables to show portfolio priorities and emphases

11. Unique Features of PREP • Upper level panelists are asked to systematically assess distinct dimensions of the 3 OMB R&D criteria • Scoring process is standardized across portfolios, transparent, & scientifically based • Therefore, PREP can provide quantitative performance assessment of portfolios of research work

12. “Portfolio” as a new concept • Portfolio as unit of analysis is new concept • Funding lines, programs, and organization of CSREES work units does not include portfolios • Use of Knowledge Areas codes for all work classification is new • Allows complex interrelated programs and funding lines to be described as they address objectives and goals of USDA

13. Portfolio Review Expert Panel (PREP) Process • Selection of high-level panelists with broad experience in topic area, reviewed for absence of conflict of interest: • University Vice-Presidents • Deans and Associate Deans • Industry Experts (Vice Presidents) • Evaluation Experts • Experts from other federal agencies

14. Portfolio Review Expert Panel (PREP) Process (cont.) • Self assessment document sent to panelists prior to meeting: • Section I: Agency and PREP Overview • Section II: Portfolio Description • Section III: Knowledge Area Descriptions • Section IV: Discussion of how portfolio meets R&D Criteria and their Dimensions • Compilation of evidentiary materials • Self assessment scoring prior to panel

15. Portfolio Review Expert Panel (PREP) Process (cont.) 3. Panelists meet for 2 ½ days: • Day 1 for orientation, short briefings by managers, along with Q&As • Day 2 for further review of documentation, discussion, voting, and recommendations • Day 3 to complete draft report containing score for PART & BPI + recommendations for portfolio improvement; debriefing by panel

16. Interim Annual Internal Review • Update self-review document • Consider recommendations from Review Panel and describe Agency responses • Used as interim preparation for next external review at the fifth year

17. 5 Dimensions of Relevance • Scope • Focus on critical needs • Identification of emerging issues • Integration of CSREES programs • Interdisciplinary integration

18. 4 Dimensions of Quality • Significance of findings & outputs • Stakeholder assessment • Alignment of portfolio with current science • Methodological rigor

19. 5 Dimensions of Performance • Portfolio productivity • Portfolio completeness • Portfolio timeliness • Agency guidance relevant to portfolio • Portfolio accountability

20. Panel Scoring Sheet • Panel scores each dimension of each of three R & D criteria using customized anchors on a 3-point scale: 3=Exceeds expectations 2=Meets expectations 1=Needs improvement

21. Panel Scoring Sheet Example: Relevance

22. Logic models • Use strategic planning and evaluation design features of logic models extensively in the self-review phase of the PREP process

23. Inputs Outputs Activities- What we do Outcomes-Impact Short term Medium Term Long Term • What we invest • Investments $ • Collaborative Ventures • Researchers • Labs & Facilities • Research: • Conduct studies • Design new techniques/ technologies • Develop models • Dissemination: • Publications • Citations • Patents • Networking What the short term results are New Knowledge, technologies, techniques, products developed and commercialized What the medium term results are Adoption/ use of new knowledge, technologies, techniques, products • What the ultimate impact(s) is • Needs met • Problems solved • Opportunitiesseized Priorities: Consider: Mission Vision Values Mandates Resources Local Dynamics Collaborators Competitors Intended Outcomes Situation Needs, Problems, and Opportunities Assumptions External Factors Evaluation Focus - Collect Data – Analyze and Interpret - Report Generic Research Logic Model Worksheet

24. Inputs Outputs Activities Participation Outcomes-Impact Short term Medium Term Long Term • What we invest • Staff • Volunteers • Time • Money • Research base • Materials • Equipment • Technology • Partners • Conduct • workshops, • meetings • Deliver • services • Develop • products, • curriculum, • resources • Train • Provide • counseling • Assess • Facilitate • Partner • Work with • Media What we do • Who we reach • Participants • Clients • Agencies • Decision- makers • CustomersSatisfaction What the short term results are Learning Awareness Knowledge Attitudes Skills Opinions Aspirations Motivations What the medium term results are Action Behavior Practice Decision-making Policies Social Action What the ultimate impact(s) is Conditions Social Economic Civic Environmental Priorities: Consider: Mission Vision Values Mandates Resources Local Dynamics Collaborators Competitors Intended Outcomes Situation Needs and Assets Symptoms versus problems Stakeholder engagement Assumptions External Factors Evaluation Focus - Collect Data – Analyze and Interpret - Report Generic Extension Logic Model Worksheet

25. Usefulness of Logic Models • Strategic Planning tool • Programmatic tool that facilitates management of programs • Guide to develop indicators for management feedback • Delineates plan for causality and attribution • Guide for monitoring and evaluation

26. Use of Logic Models in PREP • LM used for portfolio, KA, and program levels to describe plans • Panelists found logic models useful in quickly comprehending portfolios • Allowed easy description of connection of program activities to outcomes

27. Use of Logic Models in CSREES • Currently not fully integrated into programmatic activities • Will be required for state formula-funded strategic planning in FY2006 submission • Training in use of logic models has been provided to leaders

28. CSREES Marine Shrimp Virus Research Logic Model Situation Inputs Activities Outputs Outcomes Short Medium Long • Legislation: Marine Shrimp Farming Program goal is to reduce American multi-billion dollar trade deficit by developing U.S. shrimp farming industry • Problem: New and virulent viral diseases threaten the US & world shrimp farming industry • Research needed to mitigate viral threats • Research needed on high-priority opportunities to develop a sustainable U.S. commercial shrimp farming industry. • US Marine Shrimp • Farming Program • (USMSP) funds • $4 million per year • $20 million over • 5 year-period • Genetic research for: • Identification of shrimp diseases • Disease prevention • Disease resistance • Marker- • assisted • Selection for genetic shrimp stock • Develop • diagnostic • field kits for viruses • Disseminate information • via website • Breeding strategies for SPF stocks and TSV resistance • Use biosecurity • and best • management • practices • Reduce • water usage • rates • shrimp • produced • Reduce • Levels of contaminants (TSS, NH3 • and CBDO) to • lbs of shrimps • produced • New lines • Created due to • research • Reduction of disease transfer • Produce and • Distribute • Selected • stocks • Improved • economic • opportunity • for producers • & communities Basic Research • Human Capital: • Faculty • Researchers • Industry • Labs Expanded Knowledge base re Marine Shrimp Viruses Applied research Developed biosecurity protocol methods & diagnostic tests • Production: • New pedigree lines for disease resistance • New products • Marketing • Developed Marketing • Strategies for shrimp Train producers in best management practices Extension • Dissemination: • Publications, citations, patents, • Networking; industry briefs Assumptions- Joint efforts are likely to find better solutions; without the federal lead, research in this areas will not move forward; basic research in genetics is needed and will pay off in applications for combating disease; marketing research needed to complete commercialization External Factors – Shrimp prices, global trade in shrimp broodstock, new shrimp virus outbreak, intensive vs. Extensive management/production facilities, environmental activists’ reactions, transportation issues

29. Management of Forest and Rangelands Logic Model Situation Inputs Activities Outputs Outcomes Short Medium Long-term • In the 21st Century, the nation’s forest and grasslands face four threats: • Fire and fuels • Invasive Species • Loss of open space • Unmanaged recreation • The purpose of this portfolio is to rehabilitate and restore treatment priorities where risks are the greatest; prevent and contain the spread of invasive species; slow the loss of open space due to development; and manage the use of national forest for outdoor activities to increase the sustainability of our national forests and benefits to the American people • Federal • USDA • Other Federal • CSREES • State Appropriations • Determine Influence of fungal associations on fitness of sibling beetles • Determine if mycangial fungal associates change in nutrient quality of tissue • Identify genes for Dutch Elm disease resistance in American Elm. • Identify inhibitors that will reduce growth of branches and apical shoots. Extension Oregon State University Cooperative Extension Service assisted ranchers in solving problems related to rangeland ownership and management Four agencies assisted in the revitalization of coyote Creek Extension programs in cooperation with utility companies have encouraged the planting of trees around homes and buildings Trained producers in the use of best management practices Arizona Firewise Communities workshops conducted including homeowners, community leaders, fire professionals, insurance company representatives, real estate professionals and others Basic Research • Increased knowledge of rangelands for land owners and students • Invasive species • Changed beliefs that pesticides must consist of invasive or abrasive agents • Identified 2 genes for Dutch Elm disease resistance • Discovered gibberellin synthesis inhibitors paclobutrazol and flurprimidol significantly reduced elongation growth • In Arizona, new community partnerships were developed to increase fire awareness • Implemented new way to extend rangeland and environmental issues beyond a political framework • Invasive Species • Utilized two species to ward off pest. • Changes in technologies for controlling pests and diseases of urban trees, development of “designer” trees that can adapt to a wide range of urban environment • Increased overall individual and community fire awareness and action • Habitat is being restored at a low cost while biodiversity is rebounding • Recreation & wilderness • Lawsuits have been reduced which in turn saves money • A healthy, livable and sustainable urban forest ecosystem • Increased surveillance to guard against invasive species • Identified promising technologies in fire protection • Increased economic opportunities for natural resources • Human Capital: • Faculty • Researchers • Industry • Labs Applied research Invasive Species North Dakota has developed a novel pesticide which uses herbivore and preferential grazing habits of sheep and cattle to offset the damage of leafy spurge Develop semiochemical based monitoring and protective techniques to enhance integrated control strategy for pine tip moth Calculate changes in regional biogeochemistry due to urban expansion during the past several decades External Factors – Bio terrorism, invasive species, environmental disasters/concerns, high energy consumption by competitors and global trade of wood products Assumptions –; A well balanced ecosystem is an imperative backbone for the survival of our nation’s natural resources and environmental base. Natural resources play an important socio-economic role for current and future generations and need to be preserved. Source: Planning and Accountability, 2004

30. Use of Honeycomb Graphics • Displays key players; roles and coordination • Address questions of overlap and duplication across agencies/depts. • Show topic area (Portfolio and KAs) in context of world, U.S. partnerships, federal entities, USDA, and within CSREES

31. Plant Protection Portfolio KA 215 PA 214 KA 213 KA 216 KA 212 KA 211 Ecology Detection and Diagnosis Epidemiology Biology Management Prevention Economics and Safety • Disease free seed and stock for some diseases • Breeding resistant plants • Novel types of resistance genes • Marker assisted selection Prevention • Ability to identify some unculturable organisms • Characterizing take-all suppressive soils • Ecology of vectors • Genomic sequencing • Molecular communication between plants and microbes • Programmed cell death Biology and Ecology • Pathogen’s genetic fingerprint used for rapid diagnosis • Pathogenicity sequences identified • Digital diagnosis / NPDN • Seed propagation certification programs • Quarantine programs • Traditional detection methods Accomplishments Detection & Diagnosis • Understanding dynamics of spread • Understanding mechanisms of spread • Forecasting of disease based on knowledge of environmental parameters Epidemiology Areas of Science • Biological control –e.g., A. radiobacter, Trichoderma, nematodes • Chemical control (pathogen and/or vector) • Cultural practices, IPM Management • Disease loss estimates • Trade embargoes – international and interstate • Safer pesticides, reduced use through IPM Economic Sustainability and Safety • Isolate resistance genes, create resistance genes • Interfere with mechanisms of signaling, pathogen’s virulence systems • Durable resistance (understanding) Prevention • Functional genomics, proteomics, bioinformatics • Non-host resistance • Publicly accessible databases for genome-enabled biology Biology and Ecology • Culture collection and characterization, and specific DNA probes for identification • Other pathogens genetic fingerprint for rapid diagnosis • Other pathogenicity sequences • Ability to detect individuals within a microbial population • Rapid / high-throughput methods of detection (user friendly/economical) • Genomic reclassification of microbial taxonomy Detection & Diagnosis KA 212 Pathogens and Nematodes Affecting Plants • Influence of global change on pathogen spread and disease establishment • Accurate determination of disease origin Epidemiology • BC agents – establishment, mechanisms, compatibility • Alternatives to methyl bromide • Post-harvest disease • Chemical induction of resistance • Chemical resistance management Management Needs • Better understanding of impacts of diseases • Development of sustainable production practices • Development of trace-back Economic Sustainability and Safety

32. Marine Shrimp Viruses Honeycomb: a detailed program example from KAs 311-314 • Improved breeding strategies for SPF stocks and TSV • resistance • Marker assisted selection with identification of 10 ESTs for • the linkage map with 160 markers • Developed diagnostic field kits for disease prevention and • treatment • Reduced water usage rate from 4500 gal of each lb of shrimp • to 250 gal per lb of shrimp produced • Established nursery runs of 2-gram shrimp yielding up to • 6.79 kg/m2 • USMSFP produced and distributed 227,626 selected stocks • and potential broodstock and 1,211,000 Kona-line broodstock • Distributed information through publications and USMSFP • website Accomplishments KA 311 Marine Shrimp Virus PA 312 KA 313 PA 314 NAHLN KA 315 • Specific pathogen-free and genetically improved shrimp stocks • Biosecure systems including enclosed, reduced water exchange/increased water-reuse culture systems • Biosecure management practices • Co-operative industry-wide disease control strategies. Needs Research Science Areas: • Genetically improved stocks • Quantitative genetics • Gene markers and microsatellites • Advanced disease diagnostic and treatment methods • Biosecurity protocols • Effects of shrimp farming on the environment • Impact of effluent on receiving waters

33. Logic model benefits 1. Improves management 2. Facilitates evaluation 3. Facilitates accountability reporting and funding decisions

34. Our Experience Working With Agency Managers • Producing self review paper requires systematic collection and analysis of program data • Requires close collaboration between P&A staff and NPLs • Extremely demanding process given lack of readily available data

35. Our Experience Working With Expert Panels • Panelists were highly motivated, but new to the process • Panels require methodical review of process and expectations • Provide needed scores and recommendations • Panelists grateful to have support staff to assist and write panel report drafts • P&A presence during panel deliberation sessions extremely useful to clarify systems and issues

36. Summary • Self-assessment report referenced throughout the deliberation process • Inclusion of credible support data extremely important • Use of Logic Models and Portfolio graphics provide clear and concise maps of work

37. Summary (cont.) • Meaningful outcome indicators for OMB PART and BPI provided via scores from panel • Panelists provided useful recommendations • Managers have used PREP recommendations in planning, writing RFAs, budget, etc.

38. Summary (Cont.) • PREP process provided panelists with new insights in planning and accountability to spread around US • PREP improved panelists’ understanding and willingness to integrate their activities into the overall federal goals identified

39. P & A Future Directions • Network Analysis & other techniques to understand and eventually better support research system • Evaluation capacity building with agency staff and partners • Data availability: • IT & Databases • POW/ “One Solution”

40. P&A Contact Information • Telephone: 202/ 720-5623 • Fax: 202/ 720-7714 • CSREES Web site: www.csrees.usda.gov • Email: coros@csrees.usda.gov