550 likes | 560 Views
This evaluation assesses the performance of GlobAER aerosol products in 2004, focusing on aerosol optical depth (AOD) and Angstrom parameter. The study compares the data to trusted references and existing datasets, examines bias and variability, and quantifies performance using a scoring system.
E N D
Evaluationof year 2004 monthly GlobAERaerosol products Stefan Kinne
the task • an evaluation of • GlobAER 2004 global maps • for aerosol optical depth (info on amount) • for Angstrom parameter (info on size) • by ASTR (dual view, global, at best 10 per month) • by MERIS (nadir view, global, at best 1 per day) • by SEVIRI (nadir view, regional, at best 30 per day) • by a merged (ATSR / MERIS / SEVIRI) composite
the questions • how well do the data compare to trusted data references (e.g. AERONET) ? • how well do the data compare to existing data sets – even for same sensor data ? • can the performance be quantified ? • more specifically … • what are the scores of a new (outlier-resistant) method … examining • bias, spatial and temporal variability ?
the investigated properties • aerosol optical depth (AOD) • extinction along a (vertical) direction due to scattering and absorption by aerosol • here for the entire atmosphere • here for the mid-visible (0.55mm wavelength) • Angstrom parameter (Ang) • spectral dependence of AOD in the visible spectrum • small dependence (Ang ~ 0) a aerosol > 1mm size • strong decrease (Ang > 1.2) a aerosol < 0.5mm size
GlobAER 2004 maps GAa - ATSR GAs - SEVIRI GAm - MERIS GAx – merged other multi-ann. maps med – model median clim – med & aer(sun) sky – med & aer(sky) TO – TOMS other 2004 maps ATs - ATSR Swansea SEb- SEVIRI Bruxelles Mdb - MODIS deep blu MO - MODIS std coll.5 MI – MISR version 22 Ag – AVHRR, GACP Ap – AVHRR, Patmos Aer – AERONET 2004 monthly data-sets
AOD map comparisons • AOD annual maps • all available data • AOD seasonal maps • ATSR GlobAER vs Swansey • SEVIRI GloabAER vs RUIB-Bruexelles • difference maps … to a remote sensing ‘best’ composite • all available data-sets • focus on the four GloabAER products
AOD diff to ‘composite’ underestimates overestimates
quick (annual) AOD check • ATSR • underestimates in dust regions • overestimate in biomass regions • SEVIRI • severe biomass overestimates • useful over-land estimates ? • MERIS • apparent land snow cover issue • merged • not the envisioned improvement ‘-’ ’+’
Angstrom map comparisons • Angstrom annual maps • all available data • Angstrom seasonal maps • ATSR GlobAER vs Swansey • SEVIRI GloabAER vs RUIB-Bruexelles • difference maps … to a climatology (model & AERONET) • all available data-sets • focus on the four GloabAER products
Angstrom diff to ‘climatology’ . . underestimates overestimates
quick annual Angstrom check • ATSR • underestimates in tropics • overstimates in south. oceans • SEVIRI • underestimates over oceans • strong overestimates over land • MERIS • overestimates over land • merged • not the envisioned improvement ‘-’ ’+’
the SCORING challenge • quantify data performance by one number • develop a score such that contributing errors to be traceable back to • bias • spatial correlation • temporal correlation • spatial sub-scale (e.g. region) • temporal sub-scale (e.g. month, day) • make this score outlier resistant
one number ! - 0.504
info on overall bias - 0.504 sign of the bias
| 1 | is perfect …. 0 is poor - 0.504 sign of the bias the closer to absolute 1.0 … the better
product of sub-scores - 0.504 = 0.9 *- 0.7 * 0.8 temporal correlation sub-score bias sub- score spatial correlation sub-score the closer to absolute 1.0 … the better sign of the bias
spatial stratification - 0.504 = 0.9 * -0.7 * 0.8 overall score time score bias score spatial score regional surface area weights spatial sub-scale scores TRANSCOM regions
temporal stratification - 0.504 = 0.9 * -0.7 * 0.8 overall score time score bias score spatial score spatial sub-scale scores averaging in time instantaneous median data temporal sub-scale scores (e.g. month or days)
sub-score definition • each sub-score S is defined • by an error eand • by an error weight w 0.9 * -0.7 * 0.8 S = 1 – w* e time score S bias score S spatial score S spatial sub-scale scores instantaneous median data temporal sub-scale scores (e.g. month or days)
definition of errors e • S = 1 – w* e • all values for the errors e are rank - based for “time score” and “spatial score” rank correlation coefficients for data pairs are determined • e, correlation = (1- rank_correlation coeff.) /2 (correlated: e = 0, anti-correlated: e = 1) time score bias score spatial score
definition of errors e • S = 1 – w* e for “bias score” all all data-pairs are placed in a single array and ranked by value then ranks are separated according to data origin, summed and (rank-sums are) compared • e, bias= (sum1 – sum2) / (sum1 + sum2) (strong neg.bias e = -1, strong pos.bias e= +1) an example (“how does the rank bias error work ?”) • set 1: 1 7 8 value: 98 7431 rank-sum 1: 11 • set 2: 3 4 9 rank: 1 2 3 4 5 6 rank-sum 2: 10 e = (1-2)/(1+2) = (11-10)/21 ~zero a no clear bias time score bias score spatial score
definition of error weight w • S = 1 – w* e • wis a weight factor based on the inter-quartile range / median ratio • w = (75%pdf - 25%pdf) / 50%pdf … but not larger than 1.0 (w<1.0) simply put … if there is no variability an error does not matter time score bias score spatial score
scoring summary • one single score … • … without sacrificing spatial and temporal detail ! • stratification into error contribution from • bias • spatial correlation • temporal correlation • robustness against outliers • still … just one of many possible approaches • now to some applications …
questions • how did GlobAER products score? • overall ? • seasonality ? • spatial correlation ? • bias ? • in what regions ? • in what months ? • how did scores place to other retrievals … • with the same sensor (for the same year 2004) • with other sensors (for the same year 2004)
selective evaluations for year 2004 data • ..atsr Swansey ATSR • ..gaat GlobAER ATSR 3/2009 (std) • ..gaa2 GlobAER ATSR 7/2009 (filtered) • ..gaa3 GlobAER ATSR 8/2009 (test) • ..misr MISR ver.22 • how does ‘gaa2’ score ? • score diff between ‘gaa2’ and ‘misr’ • score diff between ‘gaa2’ and ‘gaat’ • score diff between ‘gaa2’ and ‘atsr’
performance vs AERONET * too few areas with scores
side by side let us compare AOD details … • what is the sign of the bias ? • what is the bias strength (median diff) ? • what is the bias error ? • what is the spatial variability error ? • what is the seasonality error ? • what is the overall error • what are selected error differences among different retrieval products?
AOD bias sign vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22 underestimate overestimate
AOD bias error vs AERONET ATSR GlobAER ATSR Swansey MODIS coll.5 MISR vers.22
AOD bias strength vs AERONET ATSR GlobAER ATSR Swansey MISR vers.22
AOD bias error vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22
AOD spatial var. Vs. AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER, filtered MISR vers.22
AOD season error vs AERONET ATSR Swansey ATSR GlobAER MISR vers.22
AOD total error vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22
AOD summary • ATSR products • ATSR is most promising (merged is poorer!) • ATSR AOD of GLOBAER • poorer than the ATSR AOD by Swansey • much poorer than MODIS or MISR • filtered version scores poorer mainly due to deterioation in spatial and temp. variability • ATSR AOD by GlobAER with filter • bias error is reasonable • spatial variability is poor • seasonality is poor
extras • score differences … for overall errors • neg. difference smaller error • pos. difference larger error • gaa2 vs atsr • gaat vs atsr • gaa2 vs gaat • gaat vs misr
gaa2 (filtered) vs atsr (swansey) black better worse green
gaat (globaer) vs atsr (swansey) black better worse green
gaa2 (filtered) vs. gaat (globaer) • b black better worse green
gaat (globaer) vs. misr better worse