1 / 47

CWFIS Process Flow Diagram - Overview

Data Source. Process. Data Storage. Deliverable. CWFIS Process Flow Diagram - Overview. View Detailed Process Flow. Weather Collection and Processing. W E A T H E R. Fire Data Warehouse. Geo Processing and Map Generation. CWFIS Web Site. D A T A S O U R C E S.

masao
Download Presentation

CWFIS Process Flow Diagram - Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Source Process Data Storage Deliverable CWFIS Process Flow Diagram - Overview View Detailed Process Flow Weather Collection and Processing W E A T H E R Fire Data Warehouse Geo Processing and Map Generation CWFIS Web Site D A T A S O U R C E S Hotspot Collection And Processing Data Marts Client Products R E M O T E S E N S I N G Satellite Image Collection and Processing

  2. Back to Overview CWFIS – Weather Collection & Processing View Detailed Process Flow Weather Station Information from MSC, WMO, NWS, Provinces Hourly and Daily Weather from MSC Data Mart Data Mart ETL Processes Provincial Daily Weather and FWI Calculations SCRIBE Forecast Data from MSC

  3. Overview CWFIS – Processes Detailed Process Flow • etl_avhrr • Process flow • -process report Etl_modis -process flow -process report etl_scribe -process flow -process report Fire M3 Geoprocessing -process flow -process report Avhrr processing -process flow -process report

  4. CWFIS – Hotspot Collection & Processing 2  Back to Overview AVHRR From NOAA ETL Processes MODIS From UMD Data Mart CWFIS Web Site FireM3 Geoprocessing ATSR From ESA AVHRR Hotspots From Satellite Imagery

  5. CWFIS – Data Warehouse & Data Mart  Back to Overview Fire Data Warehouse Annual Processes Rollups and aggregation CWFIS Data Mart Carbon Accounting Data Mart

  6. Back to Overview CWFIS – Satellite Image Collection & Processing Data Mart HRPT Antenna And Receiver ETL Processes

  7. Back to Overview Data Source Process Data Storage Deliverable CWFIS – Geoprocessing & Map Generation Data Marts Daily FWI Calculations by Station CWFIS Web Site Map and Web Image Generation Weather, FWI, FBP Grid Generation Fire Data Warehouse

  8. Back to Overview Data Source Process Data Storage Deliverable CWFIS – Web Site CWFIS Web Site Data Marts Forest Fire Conditions Interactive Mapping Forest Fire Summaries And Statistics Fire Data Warehouse General Forest Fire Information

  9. Weather Collection and Processing Canadian Weather Stations MSC metstat1 web text file Data Mart (Database tables, text files) Federal station process (can_stn2006.pl) MSC station catalogue from ftp site Data Mart (Database tables, text files) List or updates sent to NoFC upon request Provincial fire management agency Provincial station process (prov_stn.pl)

  10. Weather Collection and Processing Global Weather Stations WMO Publication 9 Volume A (Observing Stations) WMO Publication 9 Volume C (Meteorological Bulletins) Data Mart (Database tables, text files) Global station process (stnid2007.pl) NWS Global Stations Listed by WMO Number NWS Global Stations Listed by ICAO Airport Code

  11. Back to Weather Collection and Processing Hourly and Daily Weather TBUS Process Data Mart (Database and/or text files) Hourly Data Process MSC Datamart Synoptic data process Climate data process Restoration Unidata Archives Status Files

  12. Back to Timbuktu Weather Database Tables CAN_ALLSTN$Date etl_can_stn2006.pl CAN_SYNSTN$Date CAN_STN$Date create view CAN_PROVSTN$Date etl_prov_stn.pl CWFIS_STN etl_world_stn.pl USA_STN$Date CAN_HLY Blend Hly etl_hly.pl USA_HLY CWFIS_HLY CAN_HLYSYN etl_syn.pl CAN_DLY CanFwi CAN_CSWX etl_cs.pl CWFIS_FWI UAWX etl_ua.pl ADIProcess CWFIS_ADI

  13. Weather Collection and Processing FWI Calculations and ARC/INFO Maps Provincial FWI data Data Mart Weather Stations Federal weather Provincial weather and FWI Provincial FWI Process Data Mart (Database and/or text files) Federal FWI Process NoFC ftp site file Grids ARC/INFO DC, FDR Shapefiles TWN, National Atlas

  14. Weather Collection and Processing ADI Calculations and ARC/INFO Maps Data Mart Weather Stations Federal weather Data Mart (Database and/or text files) ADI Process Data Mart Rawinsonde reports from weather directory ARC/INFO Rawinsonde Decoding Grids

  15. Geoprocessing Overview Fire Danger Geoprocessing & Map Production View Process Report Automated process: cwfis.bat CWFIS web site ArcView project: c:\canada\cwfis_batch.apr (or adi_batch or fcst_batch) on Mozart1 Daily weather (Oracle cwfis_fwi) [Avenue script] PNG images: Full size (date) Full size (current) CLF size & Thumb size Automatic startup [wx_batchstart] Weather grids Forecasted weather (Postgres bt.scribe_fwi) BMP images Irfanview Forecasted wx grids SFMS ADI grids Date file Latest processing date Assemble layouts & generate images [map_maker] Text file Interpolation ADI (Oracle can_adi) FWI & FBP grids Shape file FDR shape file for map server CFFDRS grid calculation Forecasted FWI & FBP Fuel types & elevation grids Draw wind direction arrows on map (wdir or dmix) [ffcr_look_arrows] Fuel moisture grids archive on ceto FWI + hotspots map image [fwi_hotspots] Grids Fire M3 daily hotspots shape file

  16. Back to CWFIS Process Flow FireM3 – Process Flow Diagram View Fire M3 Process Report Automated process: batch_today.apr or batch.bat CWFIS runtime Fire Weather Index grid ArcView project: c:\fire3m\fire3m.apr on FM60075 [Avenue script] Make image of hotspots on FWI grid [fmmm_fwihot] Images:HPT, APT, TRI, IMG, FWIH Combine fire, smoke and cloud masks [fmmm_import] NoFC AVHRR Smoke, cloud & background TIF images Make images for web site [fmmm_images] Triple grid (fire, smoke & cloud) Date file Latest processing date Apply mask; Combine hotspots from all sources [fmmm_hpt] Calculate number of hotspots (no overlap); Generate reports [fmmm_report] University of Maryland MODIS hotspots Text files Retrieve text files via FTP [fmmm_modis_umd] Daily hotspots shape file Reports: daily & year-to-date Add today's hotspots to year-to-date file [fmmm_apt] NOAA NESDIS AVHRR hotspots Shape files Project, extract data for other organizations [fmmm_natatlas] Retrieve shape files via FTP [fmmm_fimma_ftp] Year-to-date hotspots shape file Shape files on FTP site Yesterday's year-to-date hotspots shape file Add interpolated weather, FWI & FBP for each hotspot [fmmm_resample] Buffer Hotspots --Fire Perimeter Estimates Shape file

  17. AVHRR Image Process Flow Image Processor S2-edm-fm60844 comp_order.pl AVHRR Images on fm5643 /data/fire4/hrptarch Archive91.eas chk_qtrack_dir.pl LoadLM91.eas SubmitGeocodingJobs.bat C:\hrpt\data\Archproc C:\hrpt\data\load tokyo_ftp.pl C:\geocomp_91\Rawdata qto1b PCI Geocomp TKL PCI Geocomp PCI Geocomp Weather archive /data/fire4/wx_arch C:\geocomp_91\Raw C:\geocomp_91\Geocoded C:\geocomp_91\Can03 NATAS Process NATAS data /data/fire5/dbms2/bin/mtb.pl /data/fire4/hrptarch/hrpt/Export /data/fire5/dbms2/tbus01-08/knes copytif.pl \\s2-edm-electra\pub\fire\posdata /data/fire5/dbms2/bin/clean.sh

  18. Fire M3 PDF Productionsee http://homefms.nofc.cfs.nrcan.gc.ca/FMSDocs/PeterE/pdf.html for details  Fire M3 Hotspots shape files NoFC AVHRR Background image CWFIS runtime FWI grid Copy necessary data to c:\d\canada\baby8\ ArcView: baby8.apr [Avenue: fwihot copy data] FWI grid Hotspots shape files Background image Generate PDF of satellite image ArcMap: satellite.mxd Generate PDF of daily hotspots map ArcMap: hpt.mxd Automated process: c:\d\canada\baby8\launch.bat on FM60754 Generate PDF of year-to-date hotspots map ArcMap: apt.mxd Generate PDF of hotspots-on-FWI map ArcMap: ffcrlook.mxd Optimized PDFs Hotspots on FWI Satellite image Year-to-date hotspots Daily hotspots PDFs: fwi_hot, sat_img, apt, hpt in c:\d\canada\baby8\ scram_pdf.txt SendKeys.exe Adobe Acrobat Standard

  19. Satellite Image Collection and Processing AVHRR Antenna Process Flow NOAA Polar-Orbiting Satellites Satellite TLE Ephemeris data http://www.space-track.org/perl/login.pl Manual download C:\Data Processing s2-edm-fm60844 Qtrack \\W2-edm-fm5643 NoFC HRPT Antenna

  20. CWFIS Process Flow Diagram – Detailed View Simplified View DATA COLLECTION Legend Data Source Data Warehouse: Upper Air shell MSC Data Mart Data Mart Deliverable Climate Get MSC data File Server Archive ProgControl Process CanFWI Station FWI calc DECODE Hourly Unidata Daily GEOPROCESSING O U T P U T S W E A T H E R Restore Fdr, dc Shapefiles National Atlas TWN ArcINFO Prov Wx/fwi FWI Observed PROVINCES (YK, BC, SK, ON, MB, NS, QC, PC-WBNP) ProvWx2006.pl ArcView Daily Grids (fwi, fbp, wx) Maps (for old site) Hourly forecast Forecast daily fwi Hourly (3 hr) FORECAST (CMC Scribe) (ETL_SCRIBE) DECODE, FWI CALC Maps / pngs (for website) TWN Daily Grids (fwi, fbp, wx) MOVE / Qto1B / LOAD / ARCHIVE PCI GeoComp Geocoding Hotspottextfile NoFC AVHRR  WMS CWFIS website GeoTIFFs (Fire, smoke, cloud, hotspots) Regional Images Raw Imagery (filesystem) ArcGIS (manual process) Nat. Atlas, NFIS, OMNR, PSEPC Hotspot shp files on FTP site R E M O T E S E N S I N G AVHRR (data warehouse) ETL_AVHRR NOAA / AVHRR Hotspot Processing (ArcView) FlowReport Maps & Reports (for website) ETL_MODIS MODIS (data warehouse) UMD / MODIS Generate PDFs PDF hi res maps

  21. Overview CWFIS – National Fire Data Warehouse – Collections & Products CFS collection – Agency polygon data CFS collection – Agency point data No data collected during this period Agency point data (proposed continuation) CFS collection – Hotspot point data (modis, avhrr, atsr) 1995+ CCRS provided – EO derived datasets (Landsat, modis & vgt area burned data) 2004+ Data Collections CCRS provided – EO imagery (Landsat, modis ?) 2004+ CFS collection – EO imagery (avhrr) 2002+ CFS weather data collection- daily wx (pre1950), hourly wx (1953+), upper air (2000+), dispersion index (1998+), forecast wx (2005+) FWI calculations (1970+) CFS LFDB Points 1980-1999 CFS LFDB Polygons 1980-2003 CWFIS Map Products Hotspot point data with interpolated FWI values (modis, avhrr) 1995+ National Products (derived from collections) National Burned Area Database (NBAD)- contains agency polygons & points, and EO derived datasets (provided by CCRS) CCRS provided – National Burned Area Composite (NBAC)- 2004, 2005+ (dataset compiled annually)- derived from data contained in the NBAD Daily fire perimeter estimates – polygons created by buffering hotspots past 1970 1980 1995 1999 2002 2004 2005 present future

  22. CWFIS Hardware Table 1

  23. CWFIS Hardware Table 2

  24. Ceto: 192.139.6.34Primary Function: Postgres/PostGIS database server Hardware Software Processes • Postgres 8.2.0 • PostGIS 1.2.0 • Geos 3.0.0rc2 • Proj 4.4.9 • GDAL 1.2.5 • postmaster • etl_avhrr • etl_modis • etl_scribe

  25. etl_scribe – Process Flow CWFIS Process Scribe Process Report • Initialize • Connect to DB • Update process status Load initial codes into db table Has SCRIBE data been uploaded? Select a station from the station table No Yes Obtain forecast weather, initial codes for station from table Log error message Parse SCRIBE file into scribe data structure Update process status Interpolate noon values, calculate FWI values Exit Create tables for forecast dates, insert wx data, drop old tables Exit Update database table Is the Moisture Code shapefile available? No More stations? Yes No Yes

  26. etl_modis – Process Flow CWFIS Process ETL MODIS Process Report • Initialize • Connect to DB • Update process status Read and parse record from file Update status Is FTP Server Available? No Obtain sensor id for this record Yes Log error message Exit Is the latest hotspot file available? No Format SQL and execute INSERT Exit Yes Transfer hotspot file Insert Successful? No Log error message Was file transfer Successful? Yes No More records in file? Yes No Yes

  27. etl_avhrr – Process Flow CWFIS Process ETL AVHRR Process Report • Initialize • Connect to DB • Update process status Read and parse record from file Update status Is FTP Server Available? No Obtain sensor id for this record Yes Log error message Exit Is the latest hotspot file available? No Format SQL and execute INSERT Exit Yes Transfer hotspot file Insert Successful? No Log error message Was file transfer Successful? Yes No More records in file? Yes No Yes

  28. CWFIS Process Flow Process Flow can_stn2006.pl – Process Report Process: can_stn2006.pl – gather information on federal Canadian weather stations Process File Name: can_stn2006.pl Description:Get MSC station catalogue and metstat 1 data files, parse them, and create operational station database tables Location:1.) /data/fire5/dbms2/stn/msc 2.) /data/fire9/cwfis/bin/etl_wxstn Run Time & Frequency:Occasionally, can be run at scheduled times Mode of Automation: None yet Source Code: perl Data Sources:1.) MSC metstat1 file from web site http://www.climate.weatheroffice.ec.gc.ca/prods_servs/metstat1_e.cfm 2.) MSC station catalogue file from ftp site: arcdm20.tor.ec.gc.ca/pub/doc/station_catalogue.txt Data Outputs: 1.) Database tables for station history, current hourly, and current synoptic stations. 2.) Text files spooled from each table listed above. Dependencies: None Process Documentation: Issues or Problems: 1.) Decide on scheduled update times. 2.) Still need to inspect output since some stations with different instrumentation may cause multiple entries in output. 3.) Can set up on production Linux server (ceto) any time now Steps Needed for System Integration: 1.) Utilize postgres tables for process initiation, status and error reporting. 2.) Use process naming convention: etl_can_stn.pl

  29. CWFIS Process Flow Process Flow prov_stn.pl – Process Report Process: Provincial fire management agency weather station processor Process File Name: prov_stn.pl Description:Prepare provincial weather stations data for database table creation and loading Location:1.) /data/fire7/software/bin/Provincial/stn 2.) /data/fire9/cwfis/bin/etl_wxstn Run Time & Frequency:Occasionally Mode of Automation:None yet Source Code: perl Data Sources:1.) Provincial weather station lists sent to NoFC upon request Data Outputs: 1.) Provincial weather station table and file 2.) ARC/INFO generate file for station coordinates Dependencies: See issues column Process Documentation: Issues or Problems: 1.) We must call each agency and ask for station updates. Steps Needed for System Integration: 1.) Problematic at this time: manual updates.

  30. CWFIS Process Flow Global Weather Stations – Process Report Process Flow Process: Global WMO weather station processor Process File Name: stnid007.pl Description: Get global weather station data, parse it, and create and load database tables Location:1.) /data/fire5/dbms2/stn/msc Run Time & Frequency:Occasionally, can be run at scheduled times Mode of Automation:None yet Source Code: perl Data Sources:1.) WMO Publication 9 Volume A (Observing stations) -- ftp://www.wmo.int/wmo-ddbs 2.) WMO Publication 9 Volume C (Catalogue of Meteorological Bulletins) -- ftp://www.wmo.int/wmo-ddbs/OperationalInfo/CD/VolumeC1/VolC1.txt 3.) NWS list of weather stations by WMO number -- http://weather.noaa.gov/data/nsd_bbsss.txt 4.) NWS list of weather stations by ICAO airport code -- http://weather.noaa.gov/data/nsd_cccc.txt Data Outputs: 1.) Global weather station database table and file 2.) USA station table and file? 3.) Mexico station table and file? 4.) Eurasian station table and file? Dependencies: None Process Documentation: Issues or Problems: 1.) Decide on scheduled update times 2.) Still modifiying code and adding table creation -- last udpated in 2003 3.) WMO member nations are lax about sending updated station information to WMO 4.) Can set up on Linux servers (esus and ceto) soon Steps Needed for System Integration: 1.) Utilize postgres tables for process initiation, status and error reporting. 2.) Use process naming convention: etl_stn2007.pl

  31. Weather process overview TBUS Acquisition – Process Report Process Flow Process: TBUS satellite ephemeris data acquisition Process File Name:ddwx1.sh {tbus} Description:Obtain TBUS satellite ephemeris data from the MSC data mart Location:/data/fire5/dbms2/bin Run Time & Frequency:Daily at 00:45 and 01:45 Mode of Automation:unix crontab (delphi, nofc-fm account) Source Code: Bourne shell calling perl Data Sources:MSC datamart (http:dd.weatheroffice.ec.gc.ca) Data Outputs: Raw TBUS bulletins added to the raw weather data file directory Dependencies: None Process Documentation: \\s2-edm-faunus\fireman\FMSDocs\WX\Processing\NoFCWxOp2006.doc Issues or Problems: Steps Needed for System Integration:Process is outside of CWFIS architecture.

  32. CWFIS Process Flow Dispersion Index – Process Report Process Flow Process: Lavdas Dispersion Index calculations at federal weather stations Lavdas Dispersion Index ARC/INFO maps Process File Name:1.) can_wx.sh 2.) adi_sfms.sh Description:1.) Calculate Lavdas 1986 dispersion index at federal weather stations 2.) Creates ARC/INFO grids for ADI themes Location:1.) can_wx.sh: /data/fire7/software/wxdec adidr: /datafire4/fire3/can_adi 2.) /data/fire7/software/bin Run Time & Frequency:1.) Hourly (old weather collection sh script); provides ADI values for hours 00,06,12,18 LST 2.) 04:15 and 16:15 Mode of Automation:1.) unix crontask (triton under rcarr) 2.) unix crontask (delphi under rcarr) Source Code: 1.) Bourne shell calling FORTRAN, which calls perl rawinsonde decoder 2.) ARC/INFO running AML Data Sources:1.) Weather station data from federal tables; federal hourly and upper air weather data 2.) ADI database table loaded by item 1 Data Outputs: 1.) ADI data to database table CAN_ADI Archived rawinsonde weather observation 2.) ADI grids Dependencies: 1.) Weather station data from federal tables; weather data 2.) Weather station data from federal tables, and ADI data from table loaded by item 1 Process Documentation: Issues or Problems: 1.) Rawinsonde data is not written into database tables Steps Needed for System Integration: 1a.) Move to postgres. 1b.) Process naming convention 2.) Redundant - take steps to mothball… Dispersion Index calculations at federal weather stations Dispersion Index maps using ARC/INFO

  33. CWFIS Process Flow FWI Calculations Process Report Process Flow Process: Produce FWI calculations and federal weather stations, integrate provincial FWI data, and produce ARC/INFO maps Process File Name: cansol_fwi.sh Description:a.) Gather provincial weather and FWI data which arrives at NoFC through a variety of methods b.) Run national station-based FWI calculations c.) Run ARC/INFO FWI/FBP grid and shapefile creation Location:a.) /data/fire7/software/bin/Provincial/provwx2006.pl b.) /data/fire7/software/bin/CanFwi c.) ARC/INFO project /data/fire7/canada Run Time & Frequency:Daily at 15:45 Mode of Automation:unix crontask (delphi under rcarr) Source Code:Bourne shell calling: a.) perl b.) C executable c.) ARC/INFO running AML Data Sources:a.) Provincial weather data arriving via ftp, web harvesting, email b.) Weather data in view CWFIS_WX with federal, USA, and provincial weather; plus provincial FWI data in CAN_PROVFWI table c.) National FWI in table CWFIS_FWI, plus geographic themes Data Outputs: a.) Provincial weather data in CAN_PROVWX and FWI data in CAN_PROVFWI b.) National FWI data at weather stations in table CWFIS_FWI c.) FWI/FBP grids and DC/FDR shapefiles Dependencies: a.) Provincial weather stations b.) Weather station tables, federal/USA weather data, and provincial weather/FWI data c.) FWI data from item b Process Documentation: Issues or Problems: 1.) Provincial weather/FWI needs to be arranged with NT, AB, ON, PC, NF Steps Needed for System Integration: 1.) Resolve shapefile output issue. 2.) Take steps to mothball ARC/Info processing.

  34. Weather process overview Weather Restoration Process Report Process Flow Process: Weather restoration Process File Name:dec_tk_sched.pl Description:Rerun weather collection and decoding to reduce missing reports, and report on other process status Location:\\w2-edm-fm5820 D:\Decoders\Operations\dec_tk_schedR.pl Run Time & Frequency:Hourly, but only calls restoration process every two hours from 03:30 Mode of Automation:Windows task scheduler Source Code: perl Data Sources:1.) MSC datamart (http://dd.weatheroffice.ec.gc.ca) 2.) Unidata archives (several -- should I list them all?) Data Outputs: 1.) Restored weather data written to database tables and text files 2.) Raw weather reports written to archive 3.) E-mail message to selected recipients detailing weather restoration statistics and other process status Dependencies: Hmmm Process Documentation: \\s2-edm-faunus\fireman\FMSDocs\WX\Processing\NoFCWxOp2006.doc Issues or Problems: Steps Needed for System Integration:1.) Set up new weather processing location names. 2.) Utilize postgres tables for process initiation, status and error reporting. 3.) Process naming conventions.

  35. Back to CWFIS Process Flow etl_scribe – Process Report View etl_scribe Process Flow • Process: etl_scribe: retrieve and parse MSC scribe forecast weather and insert into CWFIS fire data mart • Process File Name: etl_scribe • Source Code File Name: etl_scribe.c • Description: Hotspot processing and image production • Source Code File Location: /data/fire9/cwfis/src • Process File Location: /data/fire9/cwfis/bin • Run Time & Frequency: Once daily at 23:00 • Mode of Automation: linux cron on ceto • Source Code: C • Data Sources: Zipped SCRIBE data file uploaded by MSC to the public ftp site (/data/uploads) • Data Outputs: 72 hour forecast weather data parsed from SCRIBE data file and inserted into postgres database tables along with calculated forecast fire weather values • Dependencies: Current moisture code (FFMC, DMC, DC) shapefile • Process Documention: • Issues or Problems: - None • Steps Needed for System Integration:

  36. CWFIS Process Flow AVHRR Compositing – Process Report Process Flow Process: NoFC AVHRR imagery compositing and TIFF export Process File Name:comp_order.pl Description:Composites and archives AVHRR imagery; developed to run new products at specified times Location:\\s2-edm-fm60844 C:\geocomp_91\automaticprocessing Run Time & Frequency:Continuous Mode of Automation:Manual start Source Code: perl calling PCI EASI routines Data Sources:AVHRR imagery geocoded in item 2 Data Outputs: 1.) National composites 2.) Raw image files archived on /data/fire4/hrptarch Dependencies: Imagery from item 2 Process Documentation: Issues or Problems: Steps Needed for System Integration: 1.) Output imagery moved to CWFIS file server. 2.) Error and status reporting. 3.) Process naming convention.

  37. Weather process overview Process Flow Hourly Weather – Process Report Process: Please refer to the key  Process File Name:ddwx1.sh { can usa } ddwx1.sh {can_dwx} ddwx1.sh {can_cs} ddwx1.sh {tbus} Description:Decode Canadian or USA hourly and synoptic weather data and insert into database Decode Canadian daily weather data and insert into database Decode selected Canadian climate data and insert into database table Obtain TBUS satellite ephemeris data Location:/data/fire5/dbms2/bin Run Time & Frequency:1.) Hourly at HH:05 for Canadian data at HH-1 2.) Hourly at HH:20 for USA data at HH-1 Daily at HH:59 where HH in (00, 06, 12, 18) Hourly at HH:40 for data at HH-2 Daily at 00:45 and 01:45 Mode of Automation:unix crontab (delphi, nofc-fm account); Automatic catch-up (except TBUS) Source Code: Bourne shell calling perl Data Sources:MSC datamart (http:dd.weatheroffice.ec.gc.ca) Station database tables (except TBUS) Data Outputs: 1.) Decoded weather data file and table insertion (except TBUS) 2.) Archived raw data Dependencies: 1.) can_stn2006.pl (Canada) 2.) stnid2007.pl (USA) for creation of valid weather station tables 1.) can_stn2006.pl for creation of valid weather station tables 1.) can_stn2006.pl for creation of valid weather station tables Process Documentation: \\s2-edm-faunus\fireman\FMSDocs\WX\Processing\NoFCWxOp2006.doc Issues or Problems: Steps Needed for System Integration: 1.) Move to postgres dbms (create tables and change connection parameters). 2.) Process naming conventions. 3.) Utilize postgres tables for process initiation, status and error reporting. Hourly weather from federal stations Daily weather from federal stations Daily climate from federal station TBUS satellite ephemeris data All processes

  38. CWFIS Process Flow AVHRR Reception – Process Report Process Flow Process: AVHRR (HRPT) Reception Process File Name: Qtrack.exe Description:Receives AVHRR imagery from NoFC antenna Location:\\w2-edm-fm5643 C:\Qtrack Run Time & Frequency: Continuous Mode of Automation: Windows startup (pseudo-service) Source Code: Executable supplied by Quorum Communications Data Sources:1.) AVHRR antenna 2.) Space-Track web site for TLE ephemeris data Data Outputs: Raw AVHRR imagery written to C:\Data Dependencies: None Process Documentation: Issues or Problems: 1.) NOAA 16 noisy 2.) Dropped lines -- investigating several possible reasons 3.) Need to write script to obtain TLE ephemeris data Steps Needed for System Integration:

  39. CWFIS Process Flow AVHRR Geocoding – Process Report Process Flow Process: 1.) NoFC AVHRR imagery geocoding Process File Name:1.) SubmitGeocodingJobs.bat Description:2.) Geocoding of AVHRR imagery Location:\\s2-edm-fm60844 C:\geocomp_91\automaticprocessing Run Time & Frequency:Continuous Mode of Automation:Manual start Source Code: 1.) DOS calling perl and PCI EASI routines Data Sources: AVHRR imagery imported into PCI format in items 1a Data Outputs: Geocoded imagery Dependencies: Imagery from AVHRR Reception Process Documentation: Issues or Problems: Steps Needed for System Integration: 1.) Intermediate step for image processing. 2.) Error and status reporting.

  40. Back to CWFIS Process Flow View Fire M3 Process Flow FireM3 – Process Report • Process: FireM3 – Hotspot processing and image production • Process File Name: batch_today.apr and batch_ystdy.apr (fire3m.apr) • Description: Hotspot processing and image production • Location: s2-edm-mozart (Mozart1) • Run Time & Frequency: 4 X daily (4am, 10 am, 1pm, 6pm) • Mode of Automation: Windows scheduler calls .apr file • Source Code: Avenue scripts in .apr file • Data Sources: • AVHRR • MODIS • ATSR • Data Outputs: • - hpt and apt shape files on file server • Dependencies: • - Hotspots successfully downloaded from ftp to data warehouse • Process Documention: http://homefms.nofc.cfs.nrcan.gc.ca/FMSDocs/PeterE/firem3op.html • Issues or Problems: • - Fix bugs in FWI calc at hotspot location • Steps Needed for System Integration: • Complete transfer to Mozart • Clean up redundant data downloads (ftp to pgdb) • Clean up redundant outputs and store FWI outputs in pgdb • Standard naming convention • Error and status reporting • PDF map generation (mapserver or move to Mozart)

  41. Back to CWFIS Process Flow View etl_modis Process Flow etl_modis – Process Report • Process: etl_modis: retrieve and parse UMD MODIS hotspot data and insert into CWFIS fire data mart • Process File Name: etl_modis • Source Code File Name: etl_modis.c, modis_parse.c • Description: Hotspot ETL process • Source Code File Location: /data/fire9/cwfis/src • Process File Location: /data/fire9/cwfis/bin • Run Time & Frequency: Once daily at 13:00 • Mode of Automation: linux cron on ceto • Source Code: C • Data Sources: Hotspot text file from UMD ftp site • Data Outputs: Hotspot records loaded into postgres table ‘hotspots_modis_yyyy’ • Dependencies: None • Process Documention: • Issues or Problems: - None • Steps Needed for System Integration: Need to automate the creation of new tables for each year. • Need to run process more frequently to ensure more near real time updates.

  42. CWFIS Process Flow Fire M3 Process Flow Hotspot Process Report • Process: FireM3 – Hotspot processing and image production • Process File Name: batch_today.apr and batch_ystdy.apr (fire3m.apr) • Description: Hotspot processing and image production • Location: s2-edm-mozart (Mozart1) • Run Time & Frequency: 4 X daily (4am, 10 am, 1pm, 6pm) • Mode of Automation: Windows scheduler calls .apr file • Source Code: Avenue scripts in .apr file • Data Sources: • AVHRR • MODIS • ATSR • Data Outputs: • - hpt and apt shape files on file server • Dependencies: • - Hotspots successfully downloaded from ftp to data warehouse • Process Documention: http://homefms.nofc.cfs.nrcan.gc.ca/FMSDocs/PeterE/firem3op.html • Issues or Problems: • Fix bugs in FWI calc at hotspot location • Integrate buffer script with fireM3 • Steps Needed for System Integration: • Complete transfer to Mozart • Clean up redundant data downloads (ftp to pgdb) • Clean up redundant outputs and store FWI outputs in pgdb • Standard naming convention • Error and status reporting • PDF map generation (mapserver or move to Mozart)

  43. CWFIS Process Flow Fire M3 Process Flow Buffer Hotspots – Process Report Process: FireM3 – Hotspot Buffer Process File Name:modis_buf_prog_v10_batch.apr and modis_buf_prog_v10_yesterday.apr Description: Buffer current accumulated MODIS hotspots (.apt file) to produce updated fire perimeter estimates Location: s2-edm-mozart (Mozart2) - C:\cwfis\buf_process Run Time & Frequency: 4 X daily (4:15am, 10:15am, 1:15pm, 6:15pm) – 15 minutes after fireM3 process runs Mode of Automation: Windows scheduler calls .apr file (contains start-up script) Source Code: Avenue scripts in .apr file Data Sources:Requires daily apt file (currently on \\w2-edm-fm60075\FireM3\runtime2\aptYYmmdd.shp ) Data Outputs: w2-edm-triton\fire7\canada\Cwfis_wms\apt_buffer\.. 1) cc_apt_buf.shp - current apt buffer 2) apt_buf_prog.shp - shows daily progression 3) \yyyy\apt_buf\apt_buf_yymmdd.shp - archived 4) \yyyy\buf_prog\fgm_buf_prog_yymmdd.shp – archived Dependencies: fireM3 batch_today.apr (aptYYmmdd.shp) Process Documention: See comments in script Issues or Problems: 1) Adjust run-time to correspond with fireM3 runs 2) Integrate buffer script with fireM3 3) Adjust script to new “year” sub-directory structure Steps Needed for System Integration: 1) Look into writing code outside of ArcView (i.e. postgres/postgis) 2) Data outputs to CWFIS file server 3) Error and status reporting

  44. CWFIS Process Flow Moisture Codes at Scribe Locations – Process Report Fire M3 Process Flow Process: Forecast data – Moisture Codes at Scribe Locations Process File Name:scribe.apr Description: Sample daily moisture grids at scribe locations, used for calculating daily forecast data (if grids are not there, calculate values??) Location: will run on s2-edm-mozart (Mozart1) - C:\cwfis\scribe Run Time & Frequency: 1 X daily (10:00 pm Mode of Automation: Windows scheduler calls .apr file (contains start-up script) Source Code: Avenue scripts in .apr file Data Sources:ffmc, dmc, dc grids on w2-edm-triton\fire7\canada\runtime.. Data Outputs: w2-edm-ares\cwfis\scribe\fwi_init_yymmdd.shp Dependencies: Fwi grid calculation process (Richard) Process Documention: no documentation written up at this time -- See comments in script Issues or Problems: See “Steps Needed” below Steps Needed for System Integration: 1) Change data inputs from ArcInfo grids to ArcView grids on Mozart 2) Storage issue – network drives slow process 3) Integrate into daily grid/map generation process 4) Error and status reporting

  45. Hourly Weather – Process Report  Weather process overview Hourly weather Process Flow Process: Hourly weather from federal stations Process File Name:ddwx1.sh { can usa } Description:Decode Canadian or USA hourly weather data and insert into database Location:/data/fire5/dbms2/bin Run Time & Frequency: 1.) Hourly at HH:05 for Canadian data at HH-1 2.) Hourly at HH:20 for USA data at HH-1 Mode of Automation:unix crontab (delphi, nofc-fm account); Automatic catchup Source Code: Bourne shell calling perl Data Sources:MSC datamart (http:dd.weatheroffice.ec.gc.ca) Data Outputs: 1.) Decoded weather data file and table insertion 2.) Archived raw data Dependencies: 1.)can_stn2006.pl (Canada) 2.) stnid2007.pl (USA) for creation of valid weather station tables Process Documention: Issues or Problems: Steps Needed for System Integration: 1.)Move to postgres dbms (create tables and change connection parameters). 2. ) Process naming conventions.

  46. CWFIS Process Flow AVHRR Import – Process Report Process Flow Process: 1.) NoFC AVHRR imagery import into PCI format Process File Name:1.) chk_qtrack_dir.pl Description:1.) Retrieves raw AVHRR imagery from NoFC receiver and prepares it for geocoding Location:\\s2-edm-fm60844 C:\geocomp_91\automaticprocessing Run Time & Frequency:Continuous Mode of Automation:Manual start Source Code: 1.) perl, calling C conversion from Quorum to NOAA Level 1B formats, plus PCI GeoComp routines in EASI Data Sources:1.) Raw AVHRR imagery on fm5643 2.) Satellite ephemeris data from archive in /data/fire4/wx_arch Data Outputs: 1.) Imagery converted to Level 1B and imported into PCI format Dependencies: 1.) AVHRR Reception 2.) TBUS satellite ephemeris data from Weather Collection and Processing Process Documentation: Issues or Problems: 1.) Need conversion utility for NOAA-18 from Quorum to NOAA Level 1B formats Steps Needed for System Integration: 1.) Intermediate step for image processing. 2.) Error and status reporting.

  47. Fire danger maps  Geoprocessing overview Flow diagram Process: CWFIS actuals Process File Name: cwfis.bat Description:Produces fwi, fbp and wx grids Location:Mozart1 C:\can_sfms\cwfis.bat Run Time & Frequency:daily at 6:30 pm Mode of Automation: Windows scheduler calls batch file, which calls cwfis.apr (contains wx_batchstart start-up script) Source Code: Avenue scripts in ArcView project Data Sources: Daily weather & FWI (cwfis_fwi Oracle table) Data Outputs: ESRI grids; PNG map images; fire danger shape file Dependencies: Daily weather & FWI Process Documention: http://homefms.nofc.cfs.nrcan.gc.ca/FMSDocs/PeterE/ffcrlook.html Issues or Problems: 1. Is this duplicated by Richard's processes on ArcInfo? 2. Re-write code in C, or ArcGIS, or something else?? Steps Needed for System Integration: 1. Look at data outputs and locations 2. Storage issues 3. Error and status reporting

More Related