VALIDATION OF REMOTE SENSING CLASSIFICATIONS: a case of Balans classification

1. VALIDATION OF REMOTE SENSING CLASSIFICATIONS:a case of Balans classification Markus Törmä

2. STRUCTURE OF THESIS 1. Introduction 2. Methods for validation 3. Case: Validation of Balans classification 4. Success of validation 5. Conclusions

3. BALANS CLASSIFICATION • The goal of the project was to develop a streamlined production line for creation of land cover information using medium resolution satellite data • The production of land cover classification covering Baltic Sea drainage basin was tested

4. BALANS CLASSIFICATION Organisations: • Metria Miljöanalys • Finnish Environment Institute • Novosat Oy • GRID-Arendal • GRID-Warsaw • Swedish Meteorological and Hydrological Institute.

5. BALANS CLASSIFICATION Classes: • Artificial surfaces • Agricultural areas • Coniferous forest • Deciduous forest • Bare rock • Glaciers and perpetual snow • Other seminatural areas • Wetlands • Water bodies

6. BALANS CLASSIFICATION IRS WiFS: • Channels: RED and NIR • Spatial resolution: 188m

7. BALANS CLASSIFICATION Reference datasets: • Finnish National Land Use and Forest Classification • Swedish Terrain Type Classification • CORINE Land Cover classification • FIRS (Forest Information from Remote Sensing project) regions and stratas • Agricultural areas from Baltic Sea Region GIS, Maps and Statistical Database • Baltic Sea Drainage Basin Watershed Areas from Baltic Sea Region GIS, Maps and Statistical Database • World Atlas of Agriculture • Digital Elevation Model (DEM), slope and wetness index

8. BALANS CLASSIFICATION Classification: • Unsupervised classification of individual WiFS-images • Interpretation of clusters using reference data • Second unsupervised classification to unclassified areas • Some classes like artificial surfaces and agricultural areas are taken directly from reference data if possible

9. VALIDATION Geometric validation: • The accuracy of a geometrically rectified image Thematic validation: • The correspondence between the class label assigned to a image pixel and its true class on the field

10. GEOMETRIC VALIDATION • Validation points from 12 test areas • Residual = reference crd - classification crd • Water areas as reference data: • Finland: Digital map 1:250 000, 4 areas • Poland: Topographic map 1:100 000, 6 areas • Sweden: Topographic map 1:50 000 and 1:100 000, 2 areas

11. GEOMETRIC VALIDATION • Water areas from classification result and reference data were compared using INREC-software (VTT)

12. GEOMETRIC VALIDATION • Areal distribution of geometric control points

13. GEOMETRIC VALIDATION • Residual plots of Fuinnish, Polish and Swedish test areas

14. GEOMETRIC VALIDATION • Geometrically Balans classification was successful

15. THEMATIC VALIDATION • In order to get some idea about the mixing between classes and accuracy in the different parts of database • Check points using reference material • Classes of check points in reference material and classification result were compared • Number of check points per class was indended to be about 100 per country, but was as low as 20 in some case

16. THEMATIC VALIDATION Reference material • Finland Digital map 1:250000 Base map 1:20000 3 Landsat-7 ETM-images • Poland Corine Land Cover classification Topographic map 1:100 000 • Sweden Topographic map 1:50 000 Topographic map 1:100 000 2 Landsat-5 TM-images

17. THEMATIC VALIDATION Sampling • Aim: 100 or more reference points per class per country • Finland and Sweden: -Visual interpretation of Landsat image or map -It was tried to place reference point to an area with size more than five hectares in order to eliminate the possibity of border and mixed pixels and minimize the effect of geometric errors

18. THEMATIC VALIDATION Sampling • Poland: -GRID Warsaw -Center points of areas larger than 100 hectares from Corine landcover -Random sampling for classes artificial surfaces, agricultural areas, deciduous and coniferous forests -Classes water and wetlands were augmented using points taken from topographic map -No reference points for class seminatural areas

19. THEMATIC VALIDATION Different class-combinations were studied: • A: Original classes • B: Coniferous and deciduous forest classes are combined to one forest class • C: Agricultural and seminatural areas are combined to one class • D: One forest class and agricultural and seminatural areas are combined

20. THEMATIC VALIDATION • Following information was computed from sampling points: • Statistics of reference and classified data • Error matrix • Overall accuracy • Interpretation accuracies of classes and their mean value • Target accuracies of classes and their mean value • Average accuracies of classes and their mean value • Mapping accuracy of classes and their mean value • Kappa coefficient of classification • Tau coefficient of classification using equal a’priori probabilities for classes • 95% confidence intervals for overall, interpretation, target and average accuracies

21. THEMATIC VALIDATION • Thematic accuracy of Balans classification is not that good • Overall accuracy of original classes is 69.7% when wetlands are not included and 65.4% with wetlands • Water is classified very well • Artificial surfaces reasonably well • Otherwise from average (agricultural land) to very poor (seminatural areas) • The most problematic classes coniferous forest and seminatural areas

22. SUCCESS OF VALIDATION • Geometric validation was successful and provides unbiased view to geometric errors of Balans classification • Validation was based on check points measured from independent material (digital maps and Landsat image) • Number of check points was rather large, 294 • There were 12 test areas in Finland, Poland and Sweden so check points cover Balans classification area quite well • The effect of transformations between different coordinate systems was not evaluated

23. SUCCESS OF VALIDATION • Thematic validation was unsuccessful • Sampling design was driven by the availability of free reference data and lack of time • Effects of mixed pixels and geometric transformations were tried to minimize -visual interpretation from middle of homogeneous areas • Due to the sampling process the performed sampling is not probability sampling, more like convenience sampling.

24. SUCCESS OF VALIDATION • Finland points for deciduous forest and wetlands unreliable due to reference data • Poland points were based on CORINE which was used in classification points for seminatural areas were missing • Sweden two small test areas

25. SUCCESS OF VALIDATION Lessons: • Aim of thematic validation should be stated clearly • Sampling should be based on probability sampling • Large pixel, there is need for the validation methods for soft classifications

26. VALIDATION OF OTHER CLASSIFICATIONS • Should classification and thematic validation use hard or soft methods? • Simple, economic, unbiased and spatially comprehensive sampling method

27. VALIDATION OF OTHER CLASSIFICATIONS Spatial resolution vs. size of land cover polygons • Determine optimal spatial resolution (SR) for scene If SR of instrument smaller than optimum SR, use hard classification and validation If SR of instrument larger than optimum SR, use fuzzy classification and validation • Decide acceptable proportion of mixed pixels, i.e. pixels containing more than one land cover type If the average size of land cover polygons is known then the required spatial resolution of remote sensing instrument can be estimated.

28. VALIDATION OF OTHER CLASSIFICATIONS • Simulated effect of the size of land cover polygon to the proportion of mixed pixels • Solid line: small change causes pixel to be labeled as mixed • Dashed line: some amount of change (10%) in pixel values is tolerated

29. VALIDATION OF OTHER CLASSIFICATIONS • Sampling should be random, cover studied area and all classes • Stratified random sampling • If not cost-efficient, include clustering • Sampling points from ground visits, aerial photographs, satellite images or existing maps • More than one preson should interprete the sampling points

30. CONCLUSIONS • Geometric accuracy of Balans classification was good • Geometric validation method OK • Thematic accuracy of Balans classification poor • Due to sampling, it is dangerous to generalize thematic validation outside validation points • Thematic accuracy most likely optimistically biased