1 / 35

Building a User-Driven GEOSS: Methods to Capture, Analyze, and Prioritize User Needs

Building a User-Driven GEOSS: Methods to Capture, Analyze, and Prioritize User Needs GEO Task US-09-01a and Earth Observations Identification Lawrence Friedl, USA-NASA User Interface Committee Member US-09-01a Task Lead ISRSE-34 Symposium • Sydney, Australia 10.April .2011.

emele
Download Presentation

Building a User-Driven GEOSS: Methods to Capture, Analyze, and Prioritize User Needs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building a User-Driven GEOSS: Methods to Capture, Analyze, and Prioritize User Needs GEO Task US-09-01a and Earth Observations Identification Lawrence Friedl, USA-NASA User Interface Committee Member US-09-01a Task Lead ISRSE-34 Symposium • Sydney, Australia 10.April .2011

  2. Earth Observations NeedsOverview of Presentation Elements • This presentation discusses a document-based approach to identify observation needs, using teams of experts to: • a) review analytic methods to assess needs in documents, and b) concur on priority-setting criteria to rank the needs. • Based on results of GEO Task US-09-01a: Identifying Critical Earth Observation Priorities. • I. Approach and Meta-analysis • II. Results • III. Findings • IV. Advantages and Limitations • Discussion 2

  3. Earth Observations Needs Section I. Approach and Meta-analysis 3

  4. Earth Observations Needs Task US-09-01a GEO Task US-09-01a Objective: Establish and conduct a process for identifying critical Earth observation priorities common to many GEO societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes. GEO Societal Benefit Areas (SBA): Agriculture Disasters Health Biodiversity Ecosystems Water Climate Energy Weather 4

  5. Earth Observations Needs Task US-09-01a GEO Task US-09-01a Objective: Establish and conduct a process for identifying critical Earth observation priorities common to many GEO societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes. GEO Societal Benefit Areas (SBA): Agriculture Disasters Health Biodiversity Ecosystems Water Climate Energy Weather 5

  6. Earth Observations Needs Task US-09-01a • US-09-01a Process: Nine Steps • The process lists the steps serially, yet some are done in parallel. • Step 1: UIC Members identify Advisory Groups and Analysts for each SBA • Step 2: Determine scope of topics for the current priority-setting activity • Step 3: Identify existing documents regarding observation priorities for the SBA • Step 4: Develop analytic methods and priority-setting criteria • Step 5: Review and analyze documents for priority Earth observations needs • Step 6: Combine the information & develop a preliminary report on the priorities • Step 7: Gather feedback on the preliminary report • Step 8: Perform any additional analysis • Step 9: Complete the final report on Earth observations for the SBA 6

  7. Earth Observations Needs Task US-09-01a General Approach Document-based analysis. Examined observation needs expressed in publicly-available documents from past ~10 years. Analysis included over 1700 documents. An “Advisory Group” and an “Analyst” worked to identify documents, analyze them, and prioritize observations within each SBA. AGs involved 6-23 people from developed & developing countries that represent experts in an SBA. (Invited CoPs, IGOS, others to be on AGs). 167 AG members & 43 countries in total. Developed individual SBA reports, which specified the observation priorities for that SBA. Based on the SBA reports, the Task Team conducted a meta-analysis across the individual SBA reports to determine observations priorities common to many SBAs. The team combined and prioritized parameters from the SBA lists and prepared an over-arching report to identify “Earth observation priorities common to many SBAs.” The report includes findings, lessons learned, and recommendations. 7

  8. Earth Observations Needs Task US-09-01a Cross-SBA Analytic Methodologies Method 1: Tally of All Priorities Frequency analysis is a simple tally of the SBAs that require a given observation. (Total of 146 observations were included in this prioritization.) Methods 2&3: Weighted Sums of All Priorities Weighted frequency analysis is a weighted sum of the number of SBAs that require a given observation, taking into account the high/medium/low importance assigned by SBA Analysts. Different weighting schemes in the two methods (Same 146 parameters as Method 1.) Method 4: Top 15 Priorities by SBA This key parameters method is based on each SBA Analyst preparing a list of the “top 15” for that SBA. (Total of 99 observations were included.) Final Set: Ensemble approach across the methods. Calculated mean rank and the range of ranks for all 146 parameters across methods. Natural breaks at top 19 and top 36 parameters. 8

  9. Earth Observations Needs Section II. Results 9

  10. Earth Observations Needs Task US-09-01a SBA-specific Observation Priorities Each SBA identified a set of observation priorities as well as the 15 “most-critical” observations. Overall, there were 146 observations identified as priority needs from combining the SBA lists. The Task Team used an ensemble of 4 prioritization methods to integrate and prioritize for the Cross-SBA Analysis. Agriculture SBA Disasters SBA Energy SBA 10

  11. Earth Observations Needs Task US-09-01a 8 observations (5% of 146 total) are common to 6 or more SBAs 29 observations (20%) are common to 4 or more SBAs 100 observations (68%) are common to 2 or more SBAs 11

  12. GEO Task US-09-01a Highest Ranked Observations (#1-20) and Associated SBAs All of the 20 Highest-ranked observations are common to 4 or more SBAs All of 30 Highest-ranked obs. are common to 3 or more SBAs 3 Highest-Ranked: Precipitation, Soil Moisture, Surface Air Temperature Note: Some observations may be relevant to an SBA even though not included in the SBA’s set of priorities. * Biodiversity SBA Team did not produce a set of Earth observations priorities. 12

  13. Earth Observations Needs Section III. Findings 13

  14. Earth Observations Needs Task US-09-01a • Task Report – Findings • Precipitation Reigns the Cross-SBA Analysis • Methods Showed Agreement at Highest-Rankings • Priorities of a Single SBA May Not Be on the Cross-SBA List • Task’s Approach Produced Users’ Needs in Users’ Terminology • Articulation of Observation Needs in Documents Varied • Regional Needs Incorporated but Not Featured • Availability of Documents by Region Varied • Insufficient Information across Documents on Parameter Characteristics 14

  15. Earth Observations Needs Task US-09-01a • Task Report – Findings • Priorities of a Single SBA May Not Be on the Cross-SBA List • This approach, by design, focused on the commonality of priority observations to many SBAs. Thus, some observations of critical importance to a particular SBA do not appear in the Cross-SBA list of priority observations. • Task’s Approach Produced Users’ Needs in Users’ Terminology • Users didn’t always use terms that the Earth obs. community may be familiar with. Many of the needed observation parameters were expressed as phenomena of interest rather than technical specifications of the parameter. Overall, the demand-side, user-based approach of the task produced a rich array of observations and revealed a need for follow-on user engagement to refine parameter characteristics. • Insufficient Information across Documents on Parameter Characteristics • Some of the documents included quantitative information on the required characteristics (e.g., accuracy, latency) of critical observation parameters, while other documents lacked such information. At times it was available. However, the required observation parameter characteristics vary widely according to the user and application. 15

  16. Earth Observations Needs Task US-09-01a • Task Report – Recommendations • Gather information and engage users on specific characteristics of the priority Earth observations, especially Precipitation. • Conduct an assessment of the current and planned availability of the priority Earth observations. • GEO and/or Regional Caucuses could consider pursuing similar assessments at regional levels. • Consider additional analytic methods to gathering users’ needs and pursue an ensemble of approaches. • Prescribe the prioritization methods, SBA sub-areas, and other aspects of the SBA analyses. • Pursue broader incorporation of documents in many languages. • Continue the use of ad hoc Advisory Groups, with refinements. • Strongly consider a single organization to manage the individual SBA analyses. • Articulate an SBA’s community of users to support systematic collection of users’ needs. Follow-on Activities Process Improvements 16

  17. Earth Observations Needs Task US-09-01a • Task Report – Recommendations • GEO and/or Regional Caucuses could consider pursuing similar assessments at regional levels. • User needs vary by geographic region, and user needs unique to a single region were not likely to appear among highly-ranked observations of the Cross-SBA analysis • Pursue broader incorporation of documents in many languages. • Documents describing user needs, especially regional and national needs, exist in languages other than English, yet such documents were not discovered or were underrepresented. The Task Team recommends that future endeavors plan and provide sufficient resources for the identification of documents in many languages and for necessary translations • Articulate an SBA’s community of users to support collection of users’ needs. • Users and end-users are broad terms. Future Task Teams should develop or refine a set of User Types for each SBA. The Task Teams should ensure that the Analysts employ the User Types as guidance in collecting information and representing needs, utilizing them to conduct a gap analysis of their activities to address any bias or gaps. 17

  18. Earth Observations Needs Section IV. Advantages and Limitations 18

  19. Earth Observations Needs Task US-09-01a • Task Lead’s Comments • Task’s efforts and results represent significant contributions and first steps within GEO to articulate Earth observation priorities. GEO has documented in a transparent way how Earth observation needs have been identified, involving numerous organizations and experts. • The value of the Task’s results and cross-SBA report is at least twofold: • Provide a baseline and entry point for further engagement with end users on their needs. • Confirm any expected priority observations as backed up by an analysis of the literature. 19

  20. Earth Observations Needs Task US-09-01a • Lessons Learned • Task Approach Achieved Desired Diversity in Prioritization Methods • Variety in Analysts’ Approaches Introduced Complexities • Approach to Sponsorship of Analysts Impacted Process • Approach to Selection of SBA Sub-Areas Introduced Challenges • Advisory Groups Played Valuable Yet Variable Roles • Use of existing documents provides a level of objectivity • The ranking of an observation in the Cross-SBA list does not imply objective importance of that observation as much as commonality in need. 20

  21. Earth Observations Needs Task US-09-01a • A Document-Based Approach • Process Comments • Analytic methods to extract observations and needs • - Extraction and inference • Prioritization methods and priority-setting criteria • - Opportunity to tailor to SBA-specific issues (e.g., Daily Adjusted Life Years, World Energy Outlook) • - For individual analyses, need to be internally consistent • - For meta-analyses, need commonality and comparability across types (e.g., ordered list, tiers, grouped by some factor) • Note: The US-09-01a approach, by design and broadly-stated, was to be a natural experiment involving a variety of methods; methods were tied to the • respective SBA communities as represented by the AGs. 21

  22. Earth Observations Needs Task US-09-01a • A Document-Based Approach • Assumptions • Documents were produced in an unbiased way • Documents express needs for existing and new observations 22

  23. Earth Observations Needs Task US-09-01a • A Document-Based Approach • Advantages • Independent and objective (based on what organizations have already stated publicly) • Reflects collective needs of organizations across multiple people • Shows documented, expressed needs • Based on a ‘democratic’ principle in which all were eligible to be considered • Generate observation needs in the terms and lexicon of the respective user communities 23

  24. Earth Observations Needs Task US-09-01a • A Document-Based Approach • Limitations • Difficult to capture very-recent needs (the time aspect of the approach features long-term, sustained expressions of need); • Advisory Group member engagement and potential bias • Need for representative set of documents for all regions and SBAs • Level of specificity and technical information on needs may vary across documents; inference of needs introduces possible error 24

  25. Earth Observations Needs Task US-09-01a A Document-Based Approach p. 51 25

  26. Earth Observations Needs Task US-09-01a • Recommendations • Document-based analysis provides an objective basis to an identification of Earth observation needs • – should be part of a ensemble of analytic & prioritization methods • – should be employed to identify users’ terminology for their needs • A process involving a document-based analysis across themes should prescribe the analytic/prioritization method(s) and deliverables as well as allow for SBA-specific variants • Allocate significant time, resources, etc. to include range of documents across regions, languages • Articulate what a ‘representative’ sample comprises • – should use and internal gap analysis to ensure appropriate breadth • Allocate sufficient time for review of preliminary results by people outside of advisory groups; this review may stimulate the identification of a host of new documents) 26

  27. Earth Observations Needs Task US-09-01a Contacts & Acknowledgements Lawrence Friedl, US-09-01a Task Lead Amy Jo SWANSON, Task Coordinator Erica ZELL, Lead Analyst Adam CARPENTER, Asst. Lead Analyst Special recognition to the Analysts and Advisory Group members for the respective SBAs. Final Cross-SBA Report and Individual SBA Reports are available at the US-09-01a Task Website: http://sbageotask.larc.nasa.gov/ 27

  28. Earth Observations Needs Discussion 28

  29. Earth Observations Needs Back-up Materials 29

  30. Earth Observations Needs Task US-09-01a Advisory Groups for SBAs • Advisory Groups ranged from 6-23 members. • Sought broad international and regional representatives. • Involved GEO CoPs, former IGOS Themes, GEO Countries and GEO Participating Organizations. 30

  31. Earth Observations Needs Task US-09-01a This chart presents the 30 highest-ranked Earth observations, shown according to score in the Cross-SBA analysis. The range in ranks is also shown. Ranks are ‘inverted’ so highest score is 146. 31

  32. Earth Observations Needs Task US-09-01a General structure: An “Advisory Group” and an “Analyst” work together to identify documents, analyze them, and prioritize observations within each SBA. • Advisory Group (1 per SBA) • Functions: • - Will help to identify documents • - Comment on analytic methods and priority-setting criteria • - Review the analysts’ findings, priorities, and reports. • Involved 6-23 people from developed and developing countries that represent experts in an SBA. • Involved Communities of Practice, former IGOS Themes, GEO Countries and Participating Organizations. Analyst (1 per SBA) Functions: - Will read and analyze the documents - Develop an analytic method and priority-setting criteria - Conduct the meta-analysis to identify common priorities within a SBA. The Analyst was the primary coordinator and organizer of the activity to meet the schedules and deadlines. Interacted with and utilized the Advisory Group to vet prioritization methods and review results and reports. 32

  33. Earth Observations Needs Task US-09-01a Advisory Groups 33

  34. Earth Observations Needs Task US-09-01a A Document-Based Approach 34

  35. Earth Observations Needs Task US-09-01a Final Report Published October 2010. Delivered to GEO Plenary VII as annex to UIC Report and distributed at Plenary. Final Cross-SBA Report and Individual SBA Reports are available at the US-09-01a Task Website: http://sbageotask.larc.nasa.gov/ 35

More Related