1 / 21

Water Data Transfer in Australia

Water Data Transfer in Australia. Peter Heweston - August 2009. Ministerial Media Statement.

chyna
Download Presentation

Water Data Transfer in Australia

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Water Data Transfer in Australia Peter Heweston - August 2009

  2. Ministerial Media Statement "Australia is less than a year away from having a national water information system online, following the selection of an information technology partner for the Australian Water Resources Information System project. ... From early next year, Australians will be able to go to the Bureau's website to view, search and download water data. ... The Bureau of Meteorology is working closely with State and Territory agencies to ensure that all water data that is fed into the system is as accurate, comprehensive, consistent and up-to-date as possible." Senator Penny Wong Minister for Climate Change and Water 5 June 2009

  3. HYBOMEXP • HYBOMEXP has been in operation for nearly a year • Lots of data sent to BOM • Only recently has BOM started looking at it • Start on cycle of feeding back to Kisters and users

  4. Complex To Set Up • HYBOMEXP can be complex to set up • Many different groups of sites with different variables to be sent • long horizontal lists of encodings • QUALMAP, VARMAP • Different site list for each level variable • Risk of duplicates • Risk of missing data • Duplicated settings • FTP details in Config section • Variable mappings repeated in each section • Example

  5. A/B/C Sites • Some agencies renumber a site whenever it moves 111101A, 111101B, 111101C • Usually a few km

  6. A/B/C Sites • Composite datasource joins sites back to one virtual site • STNINI setting specified the algorithm

  7. Which A/B/C Site to BOM? • Some agencies join A/B/C back to one of the existing sites • 111101C -> 111101A,111101B,111101C • In which case, send the composite site and data to BOM • There are no composite ratings, only 111101C • There are no composite gaugings, only 111101C • Some agencies join A/B/C back to new virtual site • 111101 -> 111101A,111101B,111101C • Often no SITE, RATINGS, GAUGINGS at all for the virtual site • Send everything and let BOM sort out the mess? • They never will

  8. A/B/C Sites • Solution is to integrate A/B/C to a single site • Need to combine gaugings, ratings, sections etc • Sometimes periods overlap • 111101A and 11110B coexisted for a period • Then 111101A was shut down • Not obvious which way to go • Will need a special Hydstra job to look at periods for ratings, gaugings, sections, TS etc before merging

  9. Split Personalities • Agency A collects real time data, calls the site BW604 • Agency B collects long term archive data, and does the gauging and rating work, calls the site 232704A • How will BOM ever know they are the same site? • Lat/Long? • Who cares? • Should BW604 ratings be sent? (RATEPER for BW604 refers to 232704A tables) • No gaugings or sections for BW604 • Should 232704A ratings, gaugings, sections be duplicated in as BW604? • Notify synonyms somehow?

  10. Direction and Connection • Hydstra does not presently maintain a network diagram anywhere • Should it? • Direction and meaning of data • UrbanInflow_ML • UrbanOutflow_ML • StorageTransfer_MLd • Connectivity of sites • Upstream/Downstream • Inflow/Outflow • Travel times

  11. Subvariables • Hydstra users often use subvariables to distinguish measurements • 100.00 - downstream from weir • 100.01 - upstream from weir • Each subvariable can have its own ratings • Not gaugings • Not sections • Can't sent subvariables to BOM • Should they be a separate site? • 111101A and 11110B?

  12. Hydstra/GW

  13. Groundwater • Hydstra/GW allows one site to have many holes, one hole to have many pipes • 110.12 - > water level in hole 1 pipe 2 • WDTF doesn't support subvariables or know about holes and pipes • Which is the production pipe and which are monitoring pipes? • Hydstra/GW doesn't seem to know • Associating aquifers with pipes • Monitoring pipes are associated with an aquifer • Production pipes intersect all aquifers • Hydstra/GW doesn't seem to know

  14. Other Issues • Overlaps at BOM • Agencies sending different data to the same BOM label • 10.00 and 10.01 -> Rainfall_mm • Water level and the Geofabric • Water level is in gauge datum • Gauge datum not tied to AHD over time • Need a project to populate time tables for all sites

  15. HYBOMEXP V4 • Coming in September • More checking of INI file • Prevent duplicate variables being sent • Re-use variable mappings • Re-use FTP settings • Control over how many days of submissions to keep • Check BOM FTP size limits not exceeded • Better reporting • Send XML summary to BOM • Automatic INI updater HYFIXINI.BAT

  16. WDTF Importer • Kisters has been working with BOM to develop a quick and dirty WDTF importer • TS and SITE only • 45GB of data • 2 billion lines of PYX data • S..l..o..w to work with

  17. WDTF Import Issues • Confusion about datatrans • Cumulative rainfall sent as Rainfall_mm / instval • Nobody is using wdtf:cumulative • Evaporation - evaporation_mm / instval • Flow - Ml/d and PrecTot • Time zone issues • <wdtf:timeValuePair time="2009-04-5T01:45:00+11:00">70.86987</wdtf:timeValuePair><wdtf:timeValuePair time="2009-04-05T02:00:00+10:00">70.62244</wdtf:timeValuePair> • Duplicate data • WDTF importer does allow for delete transactions

  18. WDTF Import Issues • 400Mb file of XML with no line breaks • Technically legal XML but difficult to work with • File name problems • Files containing valid WDTF but not named correctly • Correct name but not WDTF (like Excel XML exports) • .XML files that don't contain XML • .PDF, .CSV, .TXT etc • Perl parsing performance • around 30,000 lines per second • using XMLStarlet to convert XML to PYX format • Whole job takes over 24 hours on my home quad-core Pentium

  19. WDTF Import Issues • Large Files • Hydstra low-level I/O system cannot deal with text files larger than 4GB • FAT file system cannot either, have to use NTFS • We need to enhance Hydstra in v10.1 • Handling of cumulative data • wdtf:cumulative is defined but not used so far • cannot send Rainfall_mm as InstVal

  20. Conclusion • In 1997 at HUG I said I believed data interchange was theoretically impossible • I still believe it to be true, UNLESS • We all share a common data model • We all do things the same way • We all use common codes, names and numbers • Who knows what concepts the proposed Australian Water Resources Information System will embody • You know, the one that will be ready by the end of this year • Kisters continues to engage with BOM to help make AWRIS as good as it can be • The majority of BOM data has been imported via Kisters software • Kisters staff and ex-staff are heavily involved

  21. Water Data Transfer in Australia Peter Heweston - August 2009

More Related