John Murtagh, UEL

1. Data Management forGeoinformaticsA short course on good data management for taught postgraduate students in geoinformatics and related data sciences. John Murtagh, UEL

2. Data Integration

3. Types of Data

4. qualitative data quantitative data structured data unstructured data machine-readable data

5. Qualitative data is everything that refers to the quality of something: A description of colours, texture and feel of an object. E.g. description of experiences; interview are all qualitative data.

6. Qualitative data (2) refers to forms of data collection and analysis which rely on understanding, with an emphasis on meanings rather than numerical form. It’s typically descriptive. E.g. diary accounts, open-ended questionnaires, unstructured interviews and unstructured observations.

7. Quantitative data (1) is data that refers to a number. E.g. the number of golf balls, the size, the price, a score on a test etc.

8. Quantitative data (2) usually regarded as referring to the collection and analysis of numerical data ….which can be put into categories or in rank order, or measured in units of measurement.

9. Structured & Unstructured data

10. Structured data If you want your computer to process and analyse your data, a computer has to be able to read and process the data. This means it needs to be structured and in a machine-readable form.

11. Unstructured data Unstructured has no fixed underlying structure. E.g. PDFs and scanned images may contain information which is pleasing to the human-eye as it is laid-out nicely, but they are not machine-readable.

12. Machine-readable data is data (or metadata) which is in a format that can be understood by a computer. • 2. Data file formats intended principally for machines (RDF, XML, JSON). • 1. Human-readable data that is marked up so that it can also be read by machines Examples: microformats, RDFa 2 Types

13. Data Quality

14. Types of “bad data” • Incorrect data • Inaccurate data • Business rule violations • Inconsistent data • Incomplete data • Nonintegrateddata

15. Incorrect data • For data to be correct (valid), its values must adhere to its domain (valid values). • For example, a month must be in the range of 1–12, or a person’s age must be less than 130. Taken From: ADELMAN, S., ABAI, M., & MOSS, L. T. (2005). Data strategy [...] [...]. Upper Saddle River, NJ [u.a.], Addison-Wesley.

16. Nonintegrated data • Data that has been created separately & not with the intention of future integration. • E.g. customer data can exist on 2 or more outsourced systems under different customer numbers with different spellings of the customer name & even different phone numbers or addresses. Integrating data from such systems is a challenge.

17. Inaccurate data • A data value can be correct without being accurate. For example, the city of London and web country code for France “.fr” are both accurate but when used together (such as London, France) the country is wrong because the city of London is not in France, and the accurate country code is “co.uk”

18. Inconsistent data • Uncontrolled data redundancy results in inconsistencies. Every organization is plagued with redundant and inconsistent data. • For example names or places: “Smith, David” might also sit alongside “David Smith”. London, UK and London, England.

19. Incomplete data Data that might include elements such as Names, postal code, gender, age, NHS number might also only capture haphazardly elements such as ailment, GP name, NHS capture area or even incomplete date of birth.

20. Data (Cleansing) (cleaning) (scrubbing)

21. Tools

22. Open Knowledge Foundation: School of Data (a gentle introduction to cleaning data)

23. Section 1: “Nuts and chewing gum” - looks at the the way data is presented in spreadsheets and how it might cause errors. Section 2: “The Invisible Man” is in your spreadsheet is concerned with the problems of white spaces and non-printable characters and how they affect our ability to use the data. Section 3: “Your data is a witch’s brew” deals with consistency in data entry, and how to choose the right unit and format for data.

24. Section 4: “Did you bring the wrong suitcase (again)?” is about where to put data, and how to structure it. Accompanying these sections is a step-by-step recipe for cleaning a dataset. This is an extensive, handbook-style resource which we refer to in each section. It takes a set of ‘dirty’ data and moves it through the different steps to make it ‘clean’. – See more at: http://schoolofdata.org/handbook/courses/data-cleaning/#sthash.HNzpdzyq.dpuf

25. Cleaning up

26. Sort and Filter: The basics of spreadsheets http://schoolofdata.org/handbook/courses/sort-and-filter/

27. http://schoolofdata.org/handbook/recipes/cleaning-data-with-spreadsheets/http://schoolofdata.org/handbook/recipes/cleaning-data-with-spreadsheets/

28. Possible uses of Open Refine software. Cleaning messy data: for example if you have text file with some semi-structured data, you can edit it using transformations, facets and clustering to make the data cleanly structured. Transformation of data: converting values to other formats, normalizing and denormalizing. Parsing data from web sites: OpenRefine has a URL fetch feature and jsoup HTML parser and DOM engine. Open Refine

29. Adding data to dataset by fetching it from webservices (i.e. returning json). For example can be used for geocoding addresses to geographic coordinates. • Working with Freebase: • Augmentation of datasets with data from Freebase. • Contributing data to Freebase using Schema Alignment feature. This involves reconciliation - mapping string values in cells to entities in Freebase. http://en.wikipedia.org/wiki/OpenRefine

30. Tutorials

31. Tutorial: OpenRefine/LODRefine – A Power Tool for Cleaning Data http://schoolofdata.org/category/howto/#sthash.TEXrJElh.dpuf

32. Data manipulation

33. SPSS Data manipulation tutorial:

34. MS Access Data manipulation tutorial:

35. R Data manipulation tutorial: on the following page http://www.sr.bham.ac.uk/~ajrs/R/r-manipulate_data.html

37. Gephi The following page is about Data manipulation within Gephi.

39. Primary & Secondary data • In this video Professor Innes of University of Edinburgh talks about the differences between using primary and secondary data

40. Other sessions as part of Data Management in Geoinformatics: • Data Collection • Data Management • Data Sharing Data Management for Geoinformatics by John Murtagh as part of the Jisc funded project TraD (University of East London is licensed under a Creative Commons Attribution Share Alike Licence