Model Certification

1. Model Certification Jodi Staebell Operational Director, Ecosystem Restoration Planning Center of Expertise Rock Island, IL August 7, 2009

2. Model Certification • Process to review, improve, validate analytical tools and models • Toolbox - Ensure high quality methods and tools available to enable informed decisions • Technical quality • Theory • Computational correctness • System Quality • Usability

3. Model Certification References • Information Quality Act (PL106-544) • OMB Information Quality Bulletin for Peer Review • Report of the Planning Models Improvement Task Force • Engineer Circular 1105-2-407 • Updated EC under development • Protocols for Certification of Planning Models • HQ Memo Policy Guidance on Certification of Ecosystem Output Models, Aug 08

4. Model Certification Basics • Definition - any model or analytical tool used to • Define problems and opportunities • Formulate alternatives • Evaluate effects • Support decision-making • Planning models, not engineering models • Review is cost-shared • In-house or contracted

5. Model Basics Stages of Model Development • Requirements stage • Development stage • External Testing • Implementation Model categories • Corporate • Regional/local • Commercial off-the-shelf • Models developed by others

6. Model Documentation • Provided by model proponent • Documentation includes • Background • Theory, assumptions, analytical requirements, formulas • Software/hardware, testing/validation process, • Availability of input data, usefulness to support project analysis, tech support, training • Software/spreadsheets also provided

7. Model Review Team • Planner/Formulator • Functional Field Expert(s) (internal and/or external) • Software programmer/spreadsheet auditor

8. Ecosystem Output Models Flowchart to Determine Level of Review Approved models include IWR Planning Suite and USFWS Published HSI Models ATR Team reviews model application Model certified/ approved for use? Yes No IEPR required? Detailed model review is required. Yes No Single use or multiple use model? Multiple Use Detailed model review is required. Single use of multiple use model? Goal to approve application of model on specific project Single Use Single Use ATR Team staffed and charged to conduct model review, supplement ATR team with outside experts, as appropriate Multiple Use Goal to recommend model for certification/ approval for use on future projects Model Review$60-120k7-10 months

9. Model Certification Process • Develop Model Certification Plan • Assemble Review Team (In-house/Contract) • Charge to Reviewers • Kick-off Meeting • Conduct Review • Draft Model Review Report • Meeting to discuss findings • Final Model Review Report • Revise model, as needed (lather, rinse, repeat!) • PCX recommend that HQ certify model

10. District MSC Headquarters PCX Review Group Model Review Process District request PCX Model Review • Model Documentation • Published reports • Table 2 information • Model software or spreadsheet PCX assign Model Review Manager Model Review Plan approved by HQ Model Review Plan Scope/Cost/Schedule/Method Model Review Contract Executed Scope of Work Independent Government Estimate Contractor Cost proposal Model Review Conducted Initial Findings Meeting held Draft Model Review Report Draft Review Report submitted Revised Model Documentation Model and model documentation revised as needed Final Model Review Report Final Review Report submitted Recommend model for certification/approval for use?

11. Recommend model for certification/approval for use? No Yes HQ certify model/approve for use? Yes District MSC Headquarters PCX Review Group Model Review Process (continued) Model and model documentation revised as needed or PCX memo describing model limitations submitted to MSC PCX recommend approval to HQ PCX posts approved model and documentation in PCX Toolbox

12. Scheduling and Timing Schedule • Start SOW to Notice to Proceed – 8-12 weeks • NTP to Final Model Review Report – 15-18 weeks • Revise model – depends on PDT • PCX recommendation to HQ – 4 weeks • HQ review and certify - ? Timing • Requirements and/or development stage • Identify models at (prior to) Feasibility Scoping Meeting • Initiate model review prior to Alternative Formulations Briefing

13. Model Certification Cost • ATR – 3 reviewers extra time to review model • In-house - $30-50k • Contract - $80-140k • PCX labor ~$10-15k • Model proponent labor • Prepare model documentation • Assist in Model Cert contract • Revise model and documentation

14. Model Certification Experience 3 National models certified • IWR Planning Suite • Beach-FX • HEC-FDA All published USFWS Habitat Suitability Index models approved for use Many reviews of single use models

15. ECO-PCX Model Reviews • Island Community Index Model, NAB • 3 Everglades models • Upper Miss Fish Passage Effectiveness Index • Wetlands Value Assessment, MVN • Habitat Evaluation and Assessment Tools (HEAT) ERDC • Floodplain forest and wet coastal prairie community models, SWG • Des Plaines Community HSI and HGM, LRC • Arid Streams HGM, SPL • 4 Ecological Models, MVM

16. Available Models • Ecosystem Restoration Gateway, Tools Tab • http://cw-environment.usace.army.mil/tools.cfm?CoP=Restore

18. How can the model proponent help? • Identify models early • Prepare model documentation • Test/validate model • Check software • Identify expertise needed • Be open to process • Pool funds for multiple-use models

19. Questions?