Data Preprocessing

1. Data Preprocessing Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2009

2. Knowledge Discovery (KDD) Process Knowledge Pattern Evaluation • Data mining—core of knowledge discovery process Data Mining Task-relevant Data Selection Data Warehouse Data Cleaning Data Integration Databases

3. Knowledge Process • Data cleaning – to remove noise and inconsistent data • Data integration – to combine multiple source • Data selection – to retrieve relevant data for analysis • Data transformation – to transform data into appropriate form for data mining • Data mining • Evaluation • Knowledge presentation

4. Why Preprocess the data • Image that you are a manager at ALLElectronics and have been charger with analyzing the company’s data • Then you realize: • Several of the attributes for carious tuples have no recorded value • Some information you want is not on recorded • Some values are reported as incomplete, noisy, and inconsistent • Welcome to real world!!

5. Why Data Preprocessing? • Data in the real world is dirty • incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data • e.g., occupation=“ ” • noisy: containing errors or outliers • e.g., Salary=“-10” • inconsistent: containing discrepancies in codes or names • e.g., Age=“42” Birthday=“03/07/1997” • e.g., Was rating “1,2,3”, now rating “A, B, C” • e.g., discrepancy between duplicate records

6. Why Is Data Dirty? • Incomplete data may come from • “Not applicable” data value when collected • Different considerations between the time when the data was collected and when it is analyzed. • Human/hardware/software problems

7. Why Is Data Dirty? • Noisy data (incorrect values) may come from • Faulty data collection instruments • Human or computer error at data entry • Errors in data transmission

8. Why Is Data Dirty? • Inconsistent data may come from • Different data sources • Functional dependency violation (e.g., modify some linked data) • Duplicate records also need data cleaning

9. Why Is Data Preprocessing Important? • No quality data, no quality mining results! • Quality decisions must be based on quality data • e.g., duplicate or missing data may cause incorrect or even misleading statistics. • Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse

10. Major Tasks in Data Preprocessing • Data cleaning • Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies • Data integration • Integration of multiple databases, data cubes, or files • Data transformation • Normalization and aggregation

11. Major Tasks in Data Preprocessing • Data reduction • Obtains reduced representation in volume but produces the same or similar analytical results • Data discretization • Part of data reduction but with particular importance, especially for numerical data

12. Forms of Data Preprocessing

13. Descriptive data summarization • Motivation • To better understand the data: central tendency, variation and spread • Data dispersion characteristics • median, max, min, quantiles, outliers, variance, etc.

14. Descriptive data summarization • Numerical dimensions correspond to sorted intervals • Data dispersion: analyzed with multiple granularities of precision • Boxplot or quantile analysis on sorted intervals

15. Measuring the Central Tendency • Mean • Median • Mode • Value that occurs most frequently in the data • Dataset with one, two or three modes are respectively called unimodal, bimodal, and trimodal

16. Symmetric vs. Skewed Data

17. Measuring the Central Tendency • For unimodal frequency curves that are moderately skewed, we have the following empirical relation: mean – mode = 3 * (mean - median)

18. Measuring the Dispersion of Data • Quartiles, outliers and boxplots • The median is the 50th percentile • Quartiles: Q1 (25th percentile), Q3 (75th percentile) • Inter-quartile range: IQR = Q3 –Q1 • Outlier: usually, a value higher/lower than 1.5 x IQR

19. Boxplot Analysis • Five-number summary of a distribution: Minimum, Q1, M, Q3, Maximum • Boxplot • Data is represented with a box • The ends of the box are at the first and third quartiles, i.e., the height of the box is IRQ • The median is marked by a line within the box • Whiskers: two lines outside the box extend to Minimum and Maximum

20. Boxplot Analysis

21. Histogram Analysis • Graph displays of basic statistical class descriptions • Frequency histograms • A univariate graphical method • Consists of a set of rectangles that reflect the counts or frequencies of the classes present in the given data

22. Histogram Analysis

23. Quantile Plot • Displays all of the data (allowing the user to assess both the overall behavior and unusual occurrences) • Plots quantile information • For a data xidata sorted in increasing order, fiindicates that approximately 100 fi% of the data are below or equal to the value xi

24. Quantile Plot

25. Quantile-Quantile (Q-Q) Plot • Graphs the quantiles of one univariate distribution against the corresponding quantiles of another • Allows the user to view whether there is a shift in going from one distribution to another

26. Quantile-Quantile (Q-Q) Plot

27. Scatter plot • Provides a first look at bivariate data to see clusters of points, outliers, etc • Each pair of values is treated as a pair of coordinates and plotted as points in the plane

28. Scatter plot

29. Loess Curve