240 likes | 477 Views
COL: data acquisition, storage and preservation . Roger Brugge r.brugge@reading.ac.uk www.met.reading.ac.uk/~brugge. Climatological Observers Link. Department of Meteorology University of Reading. Transmission, presentation and archiving of meteorological data, 5 Oct 2011. Outline.
E N D
COL: data acquisition, storage and preservation Roger Brugger.brugge@reading.ac.uk www.met.reading.ac.uk/~brugge Climatological Observers Link Department of MeteorologyUniversity of Reading Transmission, presentation and archiving of meteorological data, 5 Oct 2011
Outline • What is COL? • What the data are used for • Data gathering • Daily versus monthly data • Old and new methods • Data quality – station gradings • Data storage • What is stored? • How is it stored? • Data preservation • For use by COL • For use by others
What is COL? • June 1950 - An advert in Weather in June 1950 by Tom Suttie • enquiring about setting up an organisation for the exchange of weather data by amateurs • 1969 - A chance sighting of this advert by Eric Bonsor • there followed an exchange of letters, and • January 1970 - first edition of the Directory of Amateur Observers published • Site details, instrumentation and observing routines • June 1970 - first issue of the Bulletin • 48 stations reporting, describing the weather of May 1970. • Nowadays more than 330 sites each month • Aim of COL - to publish the monthly bulletin at the earliest opportunity • Achieved for over 41 years now • The COL Bulletin is now a rare source of comprehensive UK monthly weather summaries. • Membership details added to the Directory of Amateur Observers • Almost 1300 stations are now included, together with photographs of stations
Data gathering • Monthly data only • A few exceptions for a daily data table • COL cannot handle the volume of daily data that would otherwise accrue • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL >archiving • Provides members with an early snapshot • The way of the future?
Data gathering • Monthly data only • A few exceptions for a daily data table • The Bulletin • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL > archiving • Provides members with an early snapshot • The way of the future?
Data gathering • Monthly data only • A few exceptions for a daily data table • The Bulletin • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL > archiving • Provides members with an early snapshot • The way of the future?
Data quality • COL welcomes observations from everyone • Multiple stations per town • Wide variety of instrumentation and exposure • Airports, universities, institutions and the hobbyist • Instrumentation • Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges • Many good quality sites • Increasing use being made of AWS equipment • Varying costs, and also performance • Need a way to distinguish between stations that (even in the same locality) might report widely differing obs • > COL station grading system (Stephen Burt)
Buxton, Norfolk Stratfield Mortimer Elderslie
Tivington Red Wharf Bay, Anglesey Compton Martin
Data quality • COL welcomes observations from everyone • Multiple stations per town • Wide variety of instrumentation and exposure • Airports, universities, institutions and the hobbyist • Instrumentation • Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges • Increasing use being made of AWS equipment • Varying costs, and also performance • Need a way to distinguish between stations that (even in the same locality) might report widely differing obs • > COL station grading system (Stephen Burt)
Station grading system(as used in 2011) Grading system development over the years has been led by Stephen Burt
Station grade + photograph + station details = a reasonable idea of data quality
Data storage • What is stored? • Monthly summaries • A subset of the data; coverage 1940s onwards • Column-aligned text files • Easily manipulated by Excel and Fortran, etc. • Used to make quick calculations and climatological values • e.g. recently published 1981-2010 averages • All monthly data, numbers and text • Database; coverage 2006 onwards • Generated by SQL/webpage • Little-used • All back issues of the bulletin available electronically • 40th anniversary CD
Members observations – where do/could they go? • Lots of high-quality COL observations (especially grade A stations) • Effectively unused or un-archived simply because they are not reporting to UKMO • Data from these sites are likely to be lost unless special arrangements are made. Currently paper copies • traditionally, • sometimes saved in local history libraries • donated to an observer living close by • sent to the local rubbish tip (the majority) • now: they can be deposited with the Chiltern Observatory Trust • Electronic data • Monthly – some/all retained by COL • Daily – ‘delete’ key • Either way – file formats soon become outdated • A proper remote archiving/backup facility is preferable. • Might BADC have a role here? • Two suggested solutions…
Preserving members observations – Possible solution (1) • 2011: Many amateur sites are fully/partial AWS • Might consumer-AWS instrument manufacturers agree a common ‘archive file’ export standard with BADC? • to simplify the archiving process • ensure amateur data is not lost when a disk crashes or the observer dies. • If this was a file that could be exported then uploaded to a BADC site, say annually, it would take only minutes. • A mandatory metadata section should be required. • Might Davis… • who are the market leaders • who currently provide an export file format for NOAA • …take a lead on this?
Preserving members observations – Possible solution (2) • BADC to propose a standard Excel spreadsheet-type template for amateur observers to use. • This would greatly simplify the transmission and storage of amateur observations to BADC. • A mandatory metadata section should be required, perhaps including site plan/s and photographs from cardinal points. • Perhaps pilot • using the existing ‘COL archive’ template materials (hourly, daily, monthly templates have been created) • and with a voluntary participation of a handful of A-grade COL stations in 2012? • Issues: • There are as many observer-owned spreadsheet formats as observers • However, given a lead by BADC, COL members might be persuaded to modify or convert their formats
More information • This presentation – • To be placed at www.met.reading.ac.uk/~brugge • Website • www.met.reading.ac.uk/~brugge/col.html • Weather article • Brugge, R., 2010. Forty years of the Climatological Observers Link. Weather, 65, 139-143. • Email the author • r.brugge@reading.ac.uk • Bulletin – some copies available today • ‘Processed data’: an example • Burt, S.D. and R. Brugge, 2011. Climatological Averages for 1981-2010 and 2001-2010 for stations appearing in the monthly bulletin of the Climatological Observers Link. ISBN 9780956948502.