Overview of the validation of active fire products

1. Overview of the validation of active fire products Ivan Csiszar UMd

2. Topics • Active fire products • Binary yes/no detection • Sub-pixel size and temperature • Fire radiative power and energy • Validation approaches • direct validation • indirect validation (input as source – check derived parameter) • Spatial accuracy

3. What is a “fire”? • Producers (remote sensing community) • the smallest mapping/sensing unit with detectable integrated “amount” of fire • location: active fire detection • summary sub-pixel characteristic: size/temp., FRP • User community • fire of interest • larger than the “smallest” actionable fire • larger than the “smallest” non-negligible fire event either individually or aggregated in space and/or time • … and its characteristics

4. The validation process • 1. Determine “absolute” detection limits • depend on a wide range of circumstances • 2. Relate “absolute” detection limits to regional fire characteristics and user requirements

5. Validation strategy • Needs to be driven by user requirements • Metrics need to be meaningful for users • Yes/no active fire count: not a continuous variable per se • rather: probabilities of detection, detection and false alarm rates + uncertainty • Fire characteristics: continuous variables • bias + error bar • Sampling strategy • core sites – rather broadly defined target areas • stratify by fire regime

6. Active fire “yes/no” product validation • Producers’ accuracy measures • detection capabilities and false alarm rates • theoretical: radiative transfer modeling • wide range of circumstances • how accurate and realistic are they? • in-situ or high resolution remote sensing reference data • empirical: • word of mouth • visual inspection – sanity check • using in-situ or high resolution remote sensing data as reference • logistical difficulties in collecting coincident reference data • difficulties in selecting proper methodology (fire parameters, metrics, statistical model etc.) • limited set of circumstances • fusion of theoretical & empirical • anchor points to support simulation-based assessment

7. Theoretical detection envelopes from radiative transfer simulation MODIS active fire product L. Giglio

8. Empirical detection envelopes from comparison with high-resolution satellite data ASTER + MODIS grid J. Morisette

9. Ground truth • Ideally, ground truth information • is coincident with the satellite observation • includes information on all circumstances that affect detection • spatially explicit temperature field within the satellite footprint (flaming, smoldering, pre- and post-burn) • atmospheric conditions (cloudiness etc.) • etc. • Information needs are similar for the validation of “yes/no” detection and sub-pixel retrievals • We rarely or never get this

10. Collection of “ground-truth” data • In-situ (ground or aircraft) • large sample collected in collaboration with fire management agencies • coordinated effort • institutional obstacles • protocols needed – what an how is recorded • prescribed (controlled) burns • more detailed data, but very limited sample • useful for algorithm calibration, realistic mapping of conditions for simulations • hardly useful for statistical analysis

11. In-situ vs. satellite burned area maps • In-situ • location • area • start date • end date Is our active fire detection within the time bracket and within (reasonable distance of) the burned area? Siberia; red: in-situ, blue: AVHRR

12. In-situ observations

13. Collection of “ground-truth” data • Moderate of high resolution satellite sensors • opportunistic (aircraft also to some extent) • may not be optimal sensing conditions (sensor gain setting etc.) • still may not be statistically representative • easier for geostationary than for polar • prescribed (controlled) burns • very limited sample • useful for algorithm calibration, realistic mapping of conditions for simulations • hardly useful for statistical analysis • difficulties in scheduling coincident observations • less and issue for geostationary!

14. Scaling up Airborne ASTER

15. Terra and BIRD orbits

16. Orbital predictions: Jan 13, 1610 Reverse prediction from March 24, 2004 Galileo Galilei

17. Active fire “yes/no” validation • Users’ accuracy measures • depends on application and the corresponding definition of “fires of interest” • relates to local/regional fire regime • detection rates (omission errors) • false alarm rates (commission errors) • sensor/gridcell resolution! • need to have statistics of local/regional fire regimes • needs to relate to a parameter that is retrievable from satellites

18. Users’ accuracy statement Southern Africa Note: this can be another remote sensing product! The same process needs to be done at this scale. This can also be fed into simulations detection rates false alarm rates

19. Validation hierarchy Stage 1 Validation:  Product accuracy has been estimated using a small number of independent measurements obtained from selected locations and time periods and ground-truth/field program effort. Stage 2 Validation: Product accuracy has been assessed over a widely distributed set of locations and time periods via several ground-truth and validation efforts. Stage 3 Validation: Product accuracy has been assessed and the uncertainties in the product well established via independent measurements in a systematic and statistically robust way representing global conditions.

20. Validation hierarchy • Beta Data Product: • early release product, minimally validated and may still contain significant errors • available to allow users to gain familiarity with data formats and parameters • product is not appropriate as the basis for quantitative scientific publications • Provisional Data Product: • product quality may not be optimal • incremental product improvements are still occurring • general research community is encouraged to participate in the validation and QA of the product, but need to be aware that product • validation and QA is ongoing • users are urged to contact science team representatives prior to use of the data in publications • may be replaced in the archive when the validated product becomes available • Validated Product: • formally validated product, although validation is still ongoing • uncertainties are well defined • ready for use in scientific publications, and by other agencies • there may be later improved versions • earlier validated versions will be deleted from the archive after a 6 month overlap period, but code for earlier versions will be maintained indefinitely

21. Validation reporting • A distributed fire product needs to be accompanied by • a statement on its validation status • product maturity stage • quantifiable information on product accuracy, using accuracy measures that are useful for that specific user group • can be complex and overwhelming

22. Summary • Validation is a two-step process • includes several activities • fusion of empirical and theoretical approaches • Most satellite-based active fire products • have been validated up to Stage 2 • product maturity status is “Provisional” • Further work is needed to • strengthen institutional collaboration between the fire research and management communities to foster the exchange and in-situ reference data and improved satellite-based active fire products • develop validation procedures and protocols • role of GOFC/GOLD and CEOS LPV • distribution of validation datasets • develop sampling strategy • Validation: tool for inter-satellite comparison