1 / 14

An Independent Technical Review Process for Government Developed Models

An Independent Technical Review Process for Government Developed Models. HASDM As a Case Study Michael J. Gabor. Outline. Introduction JAWG HASDM Technical Review Process Lessons Learned Summary and Conclusions. Introduction. U.S. Space Command JAWG HASDM. JAWG.

brita
Download Presentation

An Independent Technical Review Process for Government Developed Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Independent Technical Review Process for Government Developed Models HASDM As a Case Study Michael J. Gabor

  2. Outline • Introduction • JAWG • HASDM • Technical Review Process • Lessons Learned • Summary and Conclusions

  3. Introduction • U.S. Space Command • JAWG • HASDM

  4. JAWG • Originally associated with SSWG • Led by USSPACE/J33 and AN • Then leadership transferred to AN • Coordinate R&D of SSA (especially space surveillance) between services • Attempted USSPACE Astro standards

  5. HASDM • DCA • Use catalog to determine common corrections that are expected to account for common atmospheric characteristics • Various approaches to DCA exist • HASDM • SSULI - Special Sensor Ultraviolet Limb Imager (NRL) • Nazarenko • EDR (Storz) • E10.7 • New solar flux proxy intended to replace F10.7 • Based on satellite EUV measurements

  6. Technical Review Process • Pre-meeting info dissemination • Pre-meeting Q&A • Briefings • Q&A • Feedback forms • Organizational memos • Feedback filtered for biases, inaccuracies, ignorance, attribution

  7. Feedback Survey • Respondent information (not used for report) • HASDM Project Comments • Assumptions • Consistency of methods • Appropriateness of tests (complete, independent) • Meaningfulness of results • Traceability of conclusions • Additional thoughts

  8. Feedback Survey • Technical Review Comments • Success of review • Prior expectations • Difference between expectations and results • Value and effectiveness of process • Recommendations to improve • Additional thoughts

  9. Review Outcome • HASDM evaluation - too much to present - see paper for examples

  10. Lessons Learned • Model development • Specific criteria need to be established to enable objective evaluations • Options should be investigated and documented first (approach, baseline, parameterization, validations) • Design tests that are independent

  11. Lessons Learned • Review process • Coordination of review needs to be improved • Participants were not sure who briefs, who pays, etc. • Insufficient time for review • Lack of formalism in process • No mechanisms for tracking resolution of criticism • After USSPACECOM demise, no accountability to process

  12. Recommendations - HASDM • E10.7 was not ready for implementation • The progress of the DCA portion of HASDM merited further evaluation • Establish evaluation criteria • Investigate other options • Conduct/Improve tests for performance evaluation and operational suitability • Perform cost/benefit analysis • Distribute as much data as possible to the research community to improve the science and allow for other ideas to mature

  13. Recommendations - Technical Review • Process should be applied new models or significant modification • Review needs to become requirement for operational implementation • Organization needs to be assigned responsibility • Needs authority, independence, and impartiality • Need mechanisms for tracking resolution

  14. Summary and Conclusions • Review process used only once for HASDM • Identified criticisms of execution in model development and highlighted areas of concern • Serves as example for future reviews • Technical review is necessary to gain community buy-in and identify loose ends • Data and technical info should be made available to industry and academia • Overcome institutional inertia obstructing team work, cooperation, and interoperability

More Related