Evaluation of year 2004 monthly GlobAER aerosol products

1. Evaluationof year 2004 monthly GlobAERaerosol products Stefan Kinne

2. the task • an evaluation of • GlobAER 2004 global maps • for aerosol optical depth (info on amount) • for Angstrom parameter (info on size) • by ASTR (dual view, global, at best 10 per month) • by MERIS (nadir view, global, at best 1 per day) • by SEVIRI (nadir view, regional, at best 30 per day) • by a merged (ATSR / MERIS / SEVIRI) composite

3. the questions • how well do the data compare to trusted data references (e.g. AERONET) ? • how well do the data compare to existing data sets – even for same sensor data ? • can the performance be quantified ? • more specifically … • what are the scores of a new (outlier-resistant) method … examining • bias, spatial and temporal variability ?

4. the investigated properties • aerosol optical depth (AOD) • extinction along a (vertical) direction due to scattering and absorption by aerosol • here for the entire atmosphere • here for the mid-visible (0.55mm wavelength) • Angstrom parameter (Ang) • spectral dependence of AOD in the visible spectrum • small dependence (Ang ~ 0) a aerosol > 1mm size • strong decrease (Ang > 1.2) a aerosol < 0.5mm size

5. GlobAER 2004 maps GAa - ATSR GAs - SEVIRI GAm - MERIS GAx – merged other multi-ann. maps med – model median clim – med & aer(sun) sky – med & aer(sky) TO – TOMS other 2004 maps ATs - ATSR Swansea SEb- SEVIRI Bruxelles Mdb - MODIS deep blu MO - MODIS std coll.5 MI – MISR version 22 Ag – AVHRR, GACP Ap – AVHRR, Patmos Aer – AERONET 2004 monthly data-sets

6. AOD map comparisons • AOD annual maps • all available data • AOD seasonal maps • ATSR GlobAER vs Swansey • SEVIRI GloabAER vs RUIB-Bruexelles • difference maps … to a remote sensing ‘best’ composite • all available data-sets • focus on the four GloabAER products

7. AOD – 2004 annual maps

8. ATSR / SEVIRI – seasonal AOD

9. AOD diff to ‘composite’  underestimates overestimates 

10. quick (annual) AOD check • ATSR • underestimates in dust regions • overestimate in biomass regions • SEVIRI • severe biomass overestimates • useful over-land estimates ? • MERIS • apparent land snow cover issue • merged • not the envisioned improvement  ‘-’ ’+’ 

11. Angstrom map comparisons • Angstrom annual maps • all available data • Angstrom seasonal maps • ATSR GlobAER vs Swansey • SEVIRI GloabAER vs RUIB-Bruexelles • difference maps … to a climatology (model & AERONET) • all available data-sets • focus on the four GloabAER products

12. Angstrom – 2004 annual maps

13. ATSR / SEVIRI – seasonal Angstr.

14. Angstrom diff to ‘climatology’ . .  underestimates overestimates 

15. quick annual Angstrom check • ATSR • underestimates in tropics • overstimates in south. oceans • SEVIRI • underestimates over oceans • strong overestimates over land • MERIS • overestimates over land • merged • not the envisioned improvement  ‘-’ ’+’ 

16. the SCORING challenge • quantify data performance by one number • develop a score such that contributing errors to be traceable back to • bias • spatial correlation • temporal correlation • spatial sub-scale (e.g. region) • temporal sub-scale (e.g. month, day) • make this score outlier resistant

17. one number ! - 0.504

18. info on overall bias - 0.504 sign of the bias

19. | 1 | is perfect …. 0 is poor - 0.504 sign of the bias the closer to absolute 1.0 … the better

20. product of sub-scores - 0.504 = 0.9 *- 0.7 * 0.8 temporal correlation sub-score bias sub- score spatial correlation sub-score the closer to absolute 1.0 … the better sign of the bias

21. spatial stratification - 0.504 = 0.9 * -0.7 * 0.8 overall score time score bias score spatial score regional surface area weights spatial sub-scale scores TRANSCOM regions

22. temporal stratification - 0.504 = 0.9 * -0.7 * 0.8 overall score time score bias score spatial score spatial sub-scale scores averaging in time instantaneous median data temporal sub-scale scores (e.g. month or days)

23. sub-score definition • each sub-score S is defined • by an error eand • by an error weight w 0.9 * -0.7 * 0.8 S = 1 – w* e time score S bias score S spatial score S spatial sub-scale scores instantaneous median data temporal sub-scale scores (e.g. month or days)

24. definition of errors e • S = 1 – w* e • all values for the errors e are rank - based for “time score” and “spatial score” rank correlation coefficients for data pairs are determined • e, correlation = (1- rank_correlation coeff.) /2 (correlated: e = 0, anti-correlated: e = 1) time score bias score spatial score

25. definition of errors e • S = 1 – w* e for “bias score” all all data-pairs are placed in a single array and ranked by value then ranks are separated according to data origin, summed and (rank-sums are) compared • e, bias= (sum1 – sum2) / (sum1 + sum2) (strong neg.bias e = -1, strong pos.bias e= +1) an example (“how does the rank bias error work ?”) • set 1: 1 7 8 value: 98 7431 rank-sum 1: 11 • set 2: 3 4 9 rank: 1 2 3 4 5 6 rank-sum 2: 10 e = (1-2)/(1+2) = (11-10)/21 ~zero a no clear bias time score bias score spatial score

26. definition of error weight w • S = 1 – w* e • wis a weight factor based on the inter-quartile range / median ratio • w = (75%pdf - 25%pdf) / 50%pdf … but not larger than 1.0 (w<1.0) simply put … if there is no variability an error does not matter time score bias score spatial score

27. scoring summary • one single score … • … without sacrificing spatial and temporal detail ! • stratification into error contribution from • bias • spatial correlation • temporal correlation • robustness against outliers • still … just one of many possible approaches • now to some applications …

28. questions • how did GlobAER products score? • overall ? • seasonality ? • spatial correlation ? • bias ? • in what regions ? • in what months ? • how did scores place to other retrievals … • with the same sensor (for the same year 2004) • with other sensors (for the same year 2004)

29. selective evaluations for year 2004 data • ..atsr Swansey ATSR • ..gaat GlobAER ATSR 3/2009 (std) • ..gaa2 GlobAER ATSR 7/2009 (filtered) • ..gaa3 GlobAER ATSR 8/2009 (test) • ..misr MISR ver.22 • how does ‘gaa2’ score ? • score diff between ‘gaa2’ and ‘misr’ • score diff between ‘gaa2’ and ‘gaat’ • score diff between ‘gaa2’ and ‘atsr’

30. performance vs AERONET * too few areas with scores

31. side by side let us compare AOD details … • what is the sign of the bias ? • what is the bias strength (median diff) ? • what is the bias error ? • what is the spatial variability error ? • what is the seasonality error ? • what is the overall error • what are selected error differences among different retrieval products?

32. AOD bias sign vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22 underestimate overestimate

33. AOD bias error vs AERONET ATSR GlobAER ATSR Swansey MODIS coll.5 MISR vers.22

34. AOD bias strength vs AERONET ATSR GlobAER ATSR Swansey MISR vers.22

35. AOD bias error vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22

36. AOD spatial var. Vs. AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER, filtered MISR vers.22

37. AOD season error vs AERONET ATSR Swansey ATSR GlobAER MISR vers.22

38. AOD total error vs AERONET ATSR GlobAER ATSR Swansey ATSR GlobAER filtered MISR vers.22

39. AOD ATSR GlobAER filtered - total error

40. AOD ATSR GlobAERfiltered - bias error

41. AOD ATSR GlobAER filtered - spatial error

42. AOD summary • ATSR products • ATSR is most promising (merged is poorer!) • ATSR AOD of GLOBAER • poorer than the ATSR AOD by Swansey • much poorer than MODIS or MISR • filtered version scores poorer mainly due to deterioation in spatial and temp. variability • ATSR AOD by GlobAER with filter • bias error is reasonable • spatial variability is poor • seasonality is poor

43. extras • score differences … for overall errors • neg. difference  smaller error • pos. difference  larger error • gaa2 vs atsr • gaat vs atsr • gaa2 vs gaat • gaat vs misr

44. gaa2 (filtered) vs atsr (swansey) black  better worse  green

45. gaat (globaer) vs atsr (swansey) black  better worse  green

46. gaa2 (filtered) vs. gaat (globaer) • b black  better worse  green

47. gaat (globaer) vs. misr  better worse 

48. AOD ATSR – GlobAER 2004

49. AOD SEVIRI – GlobAER 2004

50. AOD MERIS – GlobAER 2004