Xiwu Zhan UMBC-GEST/NASA-GSFC, Code 974.1, Greenbelt, MD Paul Houser

1. Jointly Retrieving Surface Soil Moisture from Active and Passive Microwave Observations using Cubist Data-Mining Xiwu Zhan UMBC-GEST/NASA-GSFC, Code 974.1, Greenbelt, MD Paul Houser NASA-GSFC HSB, Code 974, Greenbelt, MD Jeffrey Walker University of Melbourne, Victoria, Australia

2. OBJECTIVES • There will be high resolution (up to 1km) radar backscatter observations of land surface soil moisture from NASA ESSP Hydros mission. Radiation transfer models are usually inversed to retrieve soil moisture value. Can data-mining provide an alternative? • There are many global high resolution satellite data sets of land surface parameters that are related to soil moisture. Can we use them to derive soil moisture when radar data are not available? • What are the accuracies of these alternative soil moisture retrieval methods?

3. METHODOLOGY • Use the 1km geophysical and biophysical data fields and microwave emission and backscatter models (MEBM) from the Observation System Simulation Experiment (OSSE in Crow et al, 2004, Zhan et al, 2005) of NASA ESSP Hydros Mission; • Inverse the simulated radiometer and radar observations with the MEBM for soil moisture retrievals; • Use the Cubist data-mining tool to generate Cubist models and use the models to obtain fine resolution soil moisture retrievals • Use the update equation of the Extended Kalman Filter (EKF) to combine course resolution and fine resolution soil moisture estimations for an optimal soil moisture retrieval data product; • Compute the RMSEs of soil moisture retrievals of the three methods (INV, Cubist, & EKF) against the original soil moisture data fields.

4. Spinning 6m dish Hydros: Hydrosphere States Mission • A NASA Earth System Science Pathfinder mission; • Surface soil moisture w/ 4%vol. accuracy and Freeze/Thaw state transitions; • Revisit time: Global 3 days, boreal area 2 days • L-band (1.41GHz) Radiometer sensing 40km brightness temp. with H & V polarization; • L-band (1.26GHz) Radar measuring 1-3km backscatters with hh, vv, hv polarization; • Soil moisture products: 3km radar retrievals, 40km radiometer retrievals, 10km radar and radiometer combined retrievals and 5km 4DDA results.

5. Hydros OSSE: Data Layers 1 36km pixel 36 km TBh, TBv data from Hydros radiometer simulator 9 9km pixels 9 km soil moisture retrieval product 144 3km pixels 3 km hh, vv, hv data from Hydros radar simulator 1296 1km pixels 1 km soil moisture data from nature run

6. Radars Forward Model Radiometer Tb Forward Model 1 km Nature Run (Input: LC, ST, NDVI, Rainfall, Met data) 1km SM 1km Tsoil 1km Tskin 1km SM Hydros Instrument Simulator “Truth” Validation 1km Radiometer Tb Forward Model 1km Radars Forward Model Aggregate to 3/9/36km Red Noise Aggregate to 36km Aggregate to 3/9km White Noise White Noise 3/9/36km SM 36km Tb 3/9km s 36km Tb 3/9km s EKF Algorithm Radiometer Inversion Radar Inversion Observations From Hydros instrument simulator Background From radio- meter inversion Error Models Based on red And white noise 36km SM Tb Iteration s Iteration 36km SM Errors Tb & s EKF Algorithm 36km Radiometer Forward Model 3km Radar Forward Model Innovations 3/9/36km SM Retrieval Error Calculate optimized SM 36km SM 3/9km SM 3/9/36km SM Data Flow for Using EKF to Retrieve SM from Tb & s Observations

7. 1 km Nature Run (Input: LC, ST, NDVI, Rainfall, Met data) 1km SM 1km Tsoil 1km Tskin 1km SM Hydros Instrument Simulator “Truth” Validation 1km Radiometer Tb Forward Model 1km Radars Forward Model Aggregate to 3/9/36km Red Noise Aggregate to 36km Aggregate to 3/9km White Noise White Noise 3/9/36km SM 36km Tb 1km ndvi, Ts/s 3/9km s 36km Tb EKF Algorithm Radiometer Inversion Cubist Model Observations From Cubist model Background From radio- meter inversion Error Models Based on red And white noise 1km SM 36km SM ndvi, Ts/s Observ. Tb Iteration 36km SM Errors 1km SM EKF Algorithm 36km Radiometer Forward Model 1 km Cubist Models Innovations Calculate optimized SM 3/9/36km SM Retrieval Error 36km SM 1km SM 1/3/9/36km SM Data Flow for Using EKF to Retrieve SM from Tb & other Observations

8. Why Alternative for Radar Model? • Radar observational are not handily available for retrieving soil moisture before the launch of Hydros in 2010: spatial coverage, revisit time; • Radar radiation transfer models are not as mature as radiometer models for inversing soil moisture; • Visible/Infrared observations such as NDVI, LST and albedofrom MODIS, Landsat and future VIIRS on NPOESS are available everyday at high spatial resolutions; • The “Universal Triangle” relationships between soil moisture and the visible/infrared observations have been documented in literature for many years. NDVIs Low Soil Moisture NDVI* high SoilMoisture NDVIo To Ts T*

9. Cubist: a Data-mining Computer Tool • Cubist is used to build regression tree model of the relationships between soil moisture and its related land surface parameters such radar backscatter, or, NDVI, surface temperature and albedo; • Regression tree is similar to the decision tree classifier in that it recursively splits training samples into subsets, two at each split; • Instead of assigning class labels to the subsets, it develops a linear regression model for each of them; • Each splitting is made such that the combined residual error of the models for the two subsets is substantially lower than the residual error of the single best linear model for the samples in the two subsets, and that the combined residual error of the split is the minimum of all possible splits

10. Cubist Model Compared with Radar Model For low noise data, Cubist model of radar backscatters may reduce the RMSEs of radar model inversions by about 1-2 %v/v; Noise in data: Sigma - .5dB, Tb – 1K, ndvi – 10%, Ts-.5K, Roughness-5%, VWC-10%

11. Cubist Model Compared with Radar Model • For high noise data, Cubist model of radar backscatters could reduce the RMSEs of radar model inversions by about 3-4 %v/v; Noise in data: Sigma - 1dB, Tb – 1K, ndvi – 20%, Ts-1K, Roughness-10%, VWC-20%

12. Cubist Model Applicability/Stability • Cubist model of radar backscatters using same day or other day training data results very similar accuracy; • Cubist model based on low noise data produces almost the same accuracy as based on high noise data, and the opposite is true too.

13. Cubist Model Using Visible/IR Data • If the data noises are low, RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 3-5%v/v higher than radar backscatter model inversions.

14. Cubist Model Using Visible/IR Data • RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 1-3%v/v higher than radar backscatter model inversions based on the high noise data. Radar observations are apparently more reliable than visible/IR obs as long as a radar model is known.

15. Extended Kalman Filter for SM Retrieval Kalman filter is a statistical data assimilation technique that calculates an optimal observation correction term to the background value based on the relative magnitude of the error covariances of the observations and the background. Xa – soil moisture retrieval Xb – background SM K – Kalman gain Z – observations h(X) – obs function H – obs operator P – bg error covariance R – obs error covariance

16. EKF Application Result - 1 • EKF retrievals using Cubist model of 1km radar sigmas and 36km Tb inversion are marginally better than Cubist model estimates when the error covariance difference between Tb inversion and Cubist model is large;

17. EKF Application Result - 2 • When the error covariance difference between Tb inversion and Cubist model are smaller, the advantage of the EKF retrievals is larger (2-4% less RMSE);

18. EKF Application Result - 3 • EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are better than both Cubist model estimates and Tb inversion (2-3% less RMSE). But the Cubist model does not produce retrievals as good as using radar backscatter (4-5% larger RMSE).

19. EKF Application Result - 4 • EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are marginally better than both Cubist model estimates and Tb inversion for high noise data. Cubist model of ndvi and Ts is not as good as radar backscatter models (1-3% higher RMSE).

20. Error Distribution of SM Retrievals Low noise data, Day 155 9.1% RMSE in %v/v 3.3% 36km Tb Inversion 1km Cubist Sigma Model 5.3% 3.2% 1km EKF Tb Inv + Cubist Sigma 1km Radar Sigma Inversion

21. Error Distribution of SM Retrievals High noise data, Day 155 9.1% RMSE in %v/v 7.1% 36km Tb Inversion 1km Cubist Sigma Model 10.9% 5.9% 1km EKF Tb Inv + Cubist Sigma 1km Radar Sigma Inversion

22. SUMMARY • When radar backscatter (sigma) observations are available, a Cubist model of sigmas could be used to retrieve soil moisture with better accuracy (1-4% less RMSE) than a radar backscatter model. • The same set of equations of the Cubist model based on one set of training data may be applicable to other sets of data. Thus a Cubist model could be an alternative to a radar backscatter model based on the data we used. • EKF Data Assimilation method can combine high resolution and low resolution soil moisture estimations and improve retrieval accuracy. • Based on the NDVI and Ts used currently, a Cubist model of NDVI and Ts is not as reliable as a Cubist model of radar sigma data. The difference of RMSE could be as high as 1-5%. • However, the radar sigma inversions were obtained with the same radar backscatter model used to generate the radar backscatter data with noises added while the impacts of NDVI and Ts in the radar backscatter model may be as significant as in reality. Thus further investigations using real high resolution soil moisture, and visible/IP observational data are still needed.

23. Combining Optical/IR RS and MW RS for High Resolution Soil Moisture MODIS/VIIRS TM, SPOT, Hydros Observations of NDVI, LST, A or Sigmas Soil Moisture Truth Data from Airplane/Ground Observations AMSR-E/CMIS SMOS, Hydros, MW Observations Course Rez (20-50km) Soil Moisture Retrievals Cubist: Data Mining Tools NDVI,LST,A or Sigmas – SM Relationships EKF Data Assimilation Algorithms High Rez (30m-3km) Soil Moisture Retrievals