Report of the BERAC ARM Facility Review Panel

1. Report of the BERAC ARM Facility Review Panel Joyce E. Penner, Eugene W. Bierly, Robert Dickinson, Charles Miller1, Nelson A. Seaman1, Anne-Marie Schmoltner2, Paul Try2 1Non-ARM funded 2 Participated in 2005 review

2. Charge: Assess the effectiveness of the ARM facilities as a national user facility • How well are they serving the needs of the open (non ARM-funded) user community? • What factors are driving the costs of maintaining and operating the facilities? • Were critical recommendations from the 2005 review implemented?

3. ARM Background • ARM program funds ARM research and ARM infrastructure (sites and services) • 2004: ARM Climate Research Facility designated as a national user facility • ACRF Science Board reviews requests for field programs at the fixed sites and use of the Mobile Facility • ACRF does not fund research, but provides instrumentation, data, and expertise

4. ARM as a User Facility: External Users • The climate modeling community gains from knowledge generated by the data taken at ARM: Cloud/aerosol interactions; Convective precipitation • Non-ARM funded direct use of the facility: • Mobile facility: 3 out of last 4 users were not ARM-funded • 60% of accepted proposals are non ARM-funded P.I.’s • 1500 users; 931 use archive, 69% are non-funded • Several satellite validation exercises benefit NASA, NOAA

5. ARM as a User Facility: Possible extensions • Initiate programs to train new scientists (advanced graduate students and beginning post-docs) in the use of the ACRF • Initiate a process of outside review of the documentation of data provided in the archive

6. Cost Effectiveness of the ARM facility: Metrics • Metrics for cost effectiveness useful for DOE and OMB, but don’t tell the whole story • Scientific excellence metrics useful to capture the impact of the ACRF on the national and international research community.

7. Cost Metrics • Cost/Product: goal is $100 per product file (counted as the highest level product) (has trended down, but leveled off) • Leveraged Science Costs: Since external users not coming out of ARM, a measure of the costs of these campaigns would show broader impact

8. Science Metrics • Uptime (98%) or Number of Instrument Operation Hours • Number of Publications Citing ARM or ARM Data: 883 during 2002-2006 • Number of Proposals Received/Number of Proposals Approved (68%); would be nice to have stratified by cost • Only 17% of Mobile Facility requests approved: Strong unaccomodated desire • User and Data Use Statistics: 931 data users: need better feedback about available documentation and need error bars on products

9. Cost effectiveness and trends • Largest cost factor is labor: • Have worked to keep this cost category flat and/or declining: Savings of $6M compared to projections using 3% inflation • Have also trained staff to maintain instruments (cost savings relative to contracting this out) • Cost saving in internet services through use of higher technology

10. ACRF is approaching a period of optimal productivity • Risk that ARM Infrastructure will not be able to sustain continued growth in service to the user community under the current funding profile • Scope of ACRF responsibilities has increased with their support of climate data records: Must emphasize data quality

11. Implementation of recommendations from prior review (2005) • Management team established a formal process to address each recommendation: • Emphasis was given to: • improvement of data quality, • documentation and dissemination, • enhancement of communications to the user community

12. Web Technology, Data Products and Information • Massive improvement to the ARM web site and organization of data • A more pro-active approach to feedback from users would help • Data quality especially error characteristics could be made more accessible • Linking and cross-cutting of data sets in time could be improved • Grouping of data sets for particular problems has started and should continue

13. Other recommendations • Revival of the Instrument Development Program is probably not necessary • The criteria for deployment is certainly better enunciated especially with the Mobile Facility • Constructive outreach is being pursued: • A limited number (one to three) of signature data sets should be advertised • More focused attention could be given to applications of the world-wide, regionally diverse, collective data from the ACRF sites to global climate research • Communication of ACRF Accomplishments needs to continue

14. Other recommendations • Data quality efforts have been expanded, though no easy way to track the improvements • Management/Leadership approaches appear to be evolving to remain effective though we did not spend effort on this in the review

15. Conclusions • ACRF is effectively used by the broader scientific community • The ACRF Management has worked aggressively to decrease costs of running the ARM facility: can widen metrics to better quantify the value of the facility • ACRF management also aggressively pursued implementation of the 2005 review recommendations, though we had a few suggestions for further improvement