R. Tsheko & M. Tapela Botswana College of Agriculture

1. R. Tsheko & M. Tapela Botswana College of Agriculture Validation of the SADC THEMA Agriculture Service Products

2. Agriculture - SADC THEMA Service • Presentation outline Background Objectives Products to be validated Validation principles Sampling Field work Analysis Data Summary

3. Background • AMESD is establishing operational regional information services to support and improve the decision-making process. • In the Southern African Development Community (SADC), the theme is “Agricultural and Environmental Resource Management”. • Agriculture Service (products + services)

4. Background • RESULT AREA 2: • OPERATIONAL INFORMATION SERVICES

5. Background • RESULT AREA 2: • OPERATIONAL INFORMATION SERVICES MESA?

6. Objectives • To present validation guidelines for the Agriculture Service to NFPs • Provide a clear accuracy assessment of service products • How to monitor quality of service (timeliness, accessibility, reliability and relevence) - questionnair

7. Products to be validated • Proposal • After consultation with • the Agriculture Service developer (BDMS) • FEWSNET (Gaborone) • Former RRSU (SADC) • Four (4) products are to be validate based:- based on • Budget limitations (EUR 30.000) • Non-validated products of importance (Agriculture mask, DMP, RFEs and Crop WRSI maps) • Products where validation protocol may exist (eg. FEWSnet RFEs), moreover • The RFE product is very useful considering the fact that the SADC region has a limited network of rain gauge stations. In addition, there is a steady decline of the standard observation network, which is a strong limitation for climate related research as well as for operational agricultural monitoring.

8. Products to be validatedAP 01 Agriculture Mask • The agriculture mask outlines those areas that are dedicated to cultivation (JRC) • This crop mask has a spatial resolution of 300m • Basis for crop specific map (product) • It has not been validated

9. Products to be validatedAP14 Current Rainfall Estimate Map • In order to compensate for sparse and late reporting rain gauge stations, the agricultural service often relies upon indirect estimates of precipitation. • The RFE is a rainfall estimate of NOAA's Climate Prediction Centre currently used by FEWS-NET and several United Nations agencies such as the Food and Agriculture Organization (FAO) and World Food Programme (WFP) for agricultural monitoring in a large number of African countries. • It uses satellite imagery from the geostationary Meteosat Second Generation (MSG) and estimates convective rainfall as a function of top of cloud temperatures (the so called cold cloud duration model or (CCD) and using GTS stations for calibration. • Can not use GTS stations data for validation since they are used already in calibration

10. Satellite derived products – Remote sensingAP23 - Crop Water Requirements Satisfaction (WRSI) Index • Crop water requirement satisfaction index is the ratio of the actual evapotranspiration (AET) to the maximum evapotranspiration (MET) for a given dekad. • WRSI is an indicator of crop performance based on the availability of water to the crop during a growing season. • The map portrays WRSI values for a particular crop from the start of the growing season until this time period. It is based on the actual estimates of meteorological data to-date WRSI = f (prec, PET, WHC, Crop Type, SOS, EOS, LGP)

11. Satellite derived products – Remote sensingAP23 - Crop Water Requirements Satisfaction (WRSI) Index WRSI = f (prec, PET, WHC, Crop Type, SOS, EOS, LGP) data from NOAA, generated at EDC RFE (NOAA) FAO soils map of the world Kc (FAO)

12. Satellite derived products – DMPDry matter productivity map • DMP can be calculated by combining fAPAR, estimated from satellite imagery, with solar radiation and temperature information, as described by Monteith (1972). • The physical DMP values range between 0 and 32767 kg dry matter per hectare per day. • Cumulating DMP from the start of the season onwards provides estimate of the final dry matter production over time (kg/ha) hence “potential” production.

13. Validation principles • All the SADC-THEMA Agricultural products and services being validated now are geared towards crop condition monitoring only (the crop yield estimation shall be started in the MESA). • Therefore identifying cropped area is required now as opposed to estimating area which will be done at a later stage. • Validation is done both qualitatively and quantitatively (statistical procedures to validate). • Validation shall be done using in-situ data collected through ad-hoc, independent surveys or from independent secondary datasets. • All participating groups to validate in a similar manner so that the results can be compared across the borders.

14. Sampling RANDOM SAMPLING SYSTEMATIC SAMPLING STRATIFIED RANDOM SAMPLING STRATIFIED SYSTEMATIC UNALIGNED SAMPLING CLUSTER SAMPLING

15. Sampling Sampling grid This grid is 100kmx100km clusters within Make 10kmx10km grid Sampling unit is 1kmx1km

16. Sampling • The sample size (actual number of ground reference test samples) to be used to assess the accuracy of a map is very important. • Generally three (3) approaches could be used namely: • Binomial Probability, • Multinomial Distribution and • Rule of thumb

17. Sampling Where p is the expected percentage accuracy of the entire map, q=100-p, E is the allowable error, and Z=2 from the standard normal deviate of 1.96 for the 95% two-sided confidence level Where πi is the proportion of a population in the ith class out of k classes that has a proportion closet to 50%, bi is the desired precision (%) for this class, B is the upper (a/k)x100th percentile of the chi square (x2) distribution with 1 degree of freedom, and k is the number of classes. It is not always possible to obtain large number of random samples, therefore a balance between what is statistically sound and what is practicably attainable must be found. A good rule of thumb is to collect a minimum of 50 samples for each class in the error matrix, If the area is large (>1 million ha) or the classification has a large number of class categories, then the minimum number of samples should be increased to 75 or 100 samples per class. Note that it might be useful to take fewer samples in homogeneous classes.

18. Sampling GPS coordinates S is the minimum sample unit dimension, P is the image pixel dimension (1000m), L is geo-location accuracy in pixels (0.5 pixels), hence S=2x2 km for SPOTVGT derived products. For the crop mask, the pixel size is 300m, so the sample unit S=0.6x0.6 km Use S=1kmx1km 12 samples per 1 GPS location

19. Sampling Once the sampling sites are determined, the corresponding agriculture service product can be extracted (GPS coordinates of the sample sites are overlaid onto the satellite product using ESRI ArcMap or similar software)

20. Fieldwok • Fieldwork form in the sampling points (Dr Tapela’s presentation) • Point identification (x,y) • Date, description ie. If the point was observed or not and why • Data on land cover in and around the point (what was observed around the point, 4 or 8 neighbourhood) • Data on the environment (morphology, size) • Data on the crop • Data on the wether condition • Any other relevant information

21. Analysis • Thematic accuracy assessment: • design-based statistical inference (producer’s error, consumer’s error, overall accuracy, Kappa coefficient)

22. Analysis Thematic accuracy assessment: • design-based statistical inference (probability of detection (POD),probability o false detection (POFD), correlation (r2), root mean square error (RMSE) and multiplicative bias (BIAS)). Using independent gauge data, record detected daily precipitation events using binary (rain=1 if rainfall >=1.0mm and rain=0 (or no rain) if rainfall < 1.0mm (method by Novella and Thiaw)

23. Analysis The WRSI validation has to indirect validation based on in-situ crop condition, some qualitative crop parameters have to be chosen for this validation. (not yet finalized, using Fewsnet methodology)

24. Analysis • Thematic accuracy assessment: design-based statistical inference (correlation (r) and root mean square error (RMSE)). • A regression will then be carried out between satellite cumulative DMP and in-situ DM and the resulting r and RMSE computed to indicate the relationship.

25. Data • SANSA datasets • Your country vector maps (agric production areas) • In-situ rainfall (not used in calibration of RFEs) • TAMSAT or multisensor RFEs for estation • Other High resolution imagery (from your budget!)

26. Summary

28. The END Thank You.

29. Crop Mask - Error or confusion matrices Calculate: Row totals: Column totals: Overall accuracy:

30. Error or confusion matrices Producer’s accuracy (omission error) : User’s accuracy (commission error): Kappa coefficient of agreement:

31. Rainfall Estimates - Statistical Inference Probability of detection (POD) Probability of false detection (POFD) Correlation (r) Root mean square error (RMSE) Multiplicative bias (BIAS)).

32. DMP - Statistical Inference • Calculate Percentage Dry Matter: • Satellite cumulative DMP and in-situ DM Correlation (r) Root mean square error (RMSE)