CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008

1. CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008

2. Background Review Panel convened in response to State Water Board’s request for external review of CIWQS First Panel meeting held in May 2007 Preliminary Panel report with recommendations issued July 2007 Water Board staff to consider recommendations and demonstrate progress toward benchmarks through February 2008

3. Findings of Previous Review Seven essential recommendations: Reduce the project’s scope Restructure CIWQS’ project management Validate the system’s requirements Rebuild key constituencies Address data quality issues Produce key reports Improve user interface

4. Goals for Phase II Water Board staff will demonstrate progress by responding to Phase I recommendations Including specific actions, timelines, and additional performance measures for the coming year Panel will review progress and prepare final report to: Judge adequacy of Water Board’s response Make final recommendations for future direction of the project

5. Overview of Final Review • CIWQS remains essential for the State Water Board to fulfill its mission • Terminating funding would be counterproductive • CIWQS should be funded at an appropriate level • The program’s new direction should lead to success • We are impressed with the remarkable turnaround since May • We are particularly impressed with progress in reorganizing management and communicating with users

6. Overview (cont.) • The redefinition of project scope was appropriate and effective • We have remaining concerns with some aspects of the technical approach, rate of data cleanup, and report production

7. Original Recommendations • Reduce the scope • Restructure management • Validate system requirements • Rebuild constituency • Address data issues • Produce key reports • Improve user interface

8. Reduce the Scope • Evaluation • Achieved significant scope reduction • Increased probability of project success • Achieved buy-in from Steering Committee & users • Phased eSMR implementation plan is reasonable • Implementation through eSMR Level 3 is desirable • We concur with need for a separate federal interface • Recommendation • Consider trimming unneeded parts of the database to meet reduced project scope

9. Restructure Management • Evaluation • The new management structure positions CIWQS for success • Effective new organization with leadership in Water Quality not in DIT ties the project closer to the Board’s mission • Buy-in from senior managers for the project’s importance and new direction

10. Validate System Requirements • Evaluation • Fundamentally improved systems design & implementation • Corrected design and approach based on testing • The underlying structure can be made to work • Referential integrity issues are being addressed • Requirements better reflect user inputs & needs • Recommendations • Demonstrate a disciplined systems engineering process • Document team members’ roles, responsibilities, activities • Use test cases to demonstrate successful resolution of referential integrity issues

11. Rebuild Constituency • Evaluation • Achieved remarkable cultural change that significantly improved user communication and support • Developed effective user community structure • Users attended Panel meeting and strongly supported the project • Users strongly engaged in most areas • Issues with stormwater group illustrate cultural improvements have further to go • Recommendations • Should apply approach used for eSMR to all groups and issues

12. Address Data Issues • Evaluation • Less was accomplished than we had expected • Acknowledged the need for systematic QA/QC • Database integrity is being addressed • Some progress on correcting existing data errors • Recommendations • Devote more focused effort to QA/QC and data cleanup • Develop coherent plan that prioritizes data cleanup and identifies needed resources and timeframe • Create formal QA/QC mechanisms, including user participation • Make QA/QC an integral part of database administration and operation

13. Produce Key Reports • Evaluation • Created a mechanism to identify and prioritize key reports • Made some progress on developing canned reports • Recommendations • Demonstrate that required reports can be produced from the core CIWQS system • Increase staff’s ability to create ad hoc reports • Accelerate report production

14. Improve User Interfaces • Evaluation • Created a mechanism for input and feedback from users • Created a successful and easy-to-use SSO module • Seem to be heading in the right direction • Recommendations • Create a formal mechanism for validating and testing user interfaces • Create context-specific drop-down menus • Create context-specific online help

15. Additional Recommendations • Build a non-CIWQS interface with ICIS-NPDES • Develop a set of persistent end-to-end tests to demonstrate that the data model and processes work • Develop a deployment plan for the system and data • Fund the system at an appropriate level

16. ICIS-NPDES Issue Must Be Resolved • ICIS-NPDES will not meet State Board’s needs • There are different reporting needs for EPA and for State Board • State system should cross-check violation determinations between state and federal systems to quickly identify erroneous violations • Download ICIS-NPDES data to CIWQS as needed for State Board purposes

17. Evaluate Interface Options w/EPA • Consider discharger submission using • eDMR (file-oriented) • netDMR (interactive) • Develop test plan and case(s) including EPA DMR data download to state system • EPA has agreed to this using currently submitted data; this should be formalized quickly to take advantage of the offer

18. Develop End-to-End Testing • Develop and maintain a library of test cases with known results that can be used to test system and processes as changes are made • Should be implemented as part of all deployment plans on an ongoing basis

19. Develop Deployment Plan • Develop a deployment plan for every system update and release • Should include software, hardware, data, and quality control aspects • End-to-end testing should be integral part of the plan • Each phase of the eSMR development needs a deployment plan

20. Fund CIWQS at Appropriate Level • Panel is encouraged by progress • Believe that changes make it likely the CIWQS project will result in a system that serves the State Board’s needs for accuracy, automation, and efficiency for the future • Encourage State Board and Legislature to fund the development at levels that can maintain and accelerate development

21. Next Steps • Full report of findings and recommendations, with performance metrics, within 2 months • Panel open to revisiting progress in 9 – 12 months if asked

22. Summary • The project has made substantial improvements and is on a path to success • Management restructuring, rebuilding user constituencies, and system validation are clear successes • Additional progress needed on data cleanup, reporting, user interfaces • Additional recommendations include: • Build a non-CIWQS interface with ICIS-NPDES • Develop end-to-end tests to demonstrate all functionality • Develop comprehensive deployment plan • Fund CIWQS at an appropriate level