Global Land Cover: Approaches to Validation

1. Global Land Cover:Approaches to Validation Alan Strahler GLC2000 Meeting JRC Ispra 3/02

2. Three Methods/Approaches • Statistical Approaches (After S. Stehman) • Sampling a map using design-based inference to make accuracy statements about the map with known precision • Characterizing a classification process using model-based inference to estimate the accuracy of individual pixel labels • Confidence-Building Measures • Carrying out confidence-building measures includes making studies or comparisons without a firm statistical basis that provide confidence in the map

3. Design-Based Inference • Definition • Sampling to infer characteristics of a finite population, such as the pixels in a land cover map • Probability Sampling • Sampled units are drawn with known probabilities • Example: Random or stratified random sampling • Consistency Criterion in Estimation • An estimator of a population parameter must equal the population parameter if the sample size includes the entire population

4. Design-Based Inference, Cont. • Consistent estimators include: • Proportion of pixels correctly classified • User’s Accuracy: • Given that a pixel is mapped as A, what is the probability that it actually is A on the ground? • Producer’s Accuracy: • Given that a pixel is actually of class A, what is the probability that it is mapped correctly? • Confusion Matrix • The confusion matrix is the primary tool used to find consistent estimators • Diagonals count matches and marginal totals count number of pixels sampled

5. Problems in Design-Based Inference • “Ground Truth” • Determination of the “correct” class for a sampled pixel is not without error • Photointerpretation errors occur when fine-scale imagery is used instead of ground visits • Misregistration errors—the wrong location is visited or viewed at higher resolution • Equivocal Classification Schemes • Classes may not be mutually exclusive or be difficult to resolve • Example: Permanent wetland may also be forest (IGBP). Are both labels correct? • Classes may not be well defined • Example: What is a golf course? Is it agriculture? Grassland? Urban?

6. Problems in Design-Based Inference, Cont. • “Correctness” of Match and Mismatch • Some errors are worse than others—e.g., open shrubland vs. closed shrubland may be minor, while forest classified as water may be major • Leads to fuzzy agreement measures as better indicators of map utility • Mixed Pixels • Ground truth pixels may contain multiple classes • Which label is correct? • Leads to fuzzy confusion matrixes • Map Comparisons • Given their error structures, how do we conclude that two maps are different? • If they are different, which one is more accurate?

7. Model-Based Inference • Focuses on the classification process, not the map • E.g., Which classifier works better? • Maps as realizations of a classification process that makes random errors • Reliability Measures • Parameters that are inferred from the classification process • E.g., maximum likelihood classification gives the probability that a pixel belongs to a particular class • Can be mapped and summarized to provide information about the “quality” of a map

8. Confidence-Building Measures • “Looks good!” Reconnaissance Measures • Map conforms well to regional landscape attributes—mountains, valleys, agricultural regions, etc. • Spatial structure is sensible, not salt-and-pepper noise or excessively smooth • Land-water boundaries are clear, indicating good registration of input data • Free of major glitches, such as cities in the Sahara • Ancillary Comparisons • Does the classifier’s output conform to the general patterns of land cover documented in other datasets or maps? • Systematic Assessment • Qualitative assessment of map accuracy in a systematic (wall-to-wall) fashion

9. Summary • Design-based inference provides statements of accuracy with known precision at highest cost • Model-based inference characterizes the accuracy of the map-making process at lesser cost • Confidence-building measures assess map quality at low cost Validation can, and should, rely on all three approaches.