Basic Data Collection and Analysis

1. Basic Data Collection and Analysis TAU Breakfast Session 14 April 2011 Fia van Rensburg

2. Purpose To highlight aspects with regard to monitoring, data collection and data analysis that are relevant to TAU on an operational level To build capacity of support staff within TAU to perform basic monitoring, data collection and data analysis functions To encourage TAU support staff to request assistance form the KM unit to advise on monitoring, data collection and data analysis

3. What this presentation is not about This presentation does not give detailed information on RBM It is not a technical lecture on research methodology and statistics

4. What is happening here?

5. What is happening in the picture? We can see different things in this picture: how many children; how many girls and boys; colours of their clothes; colours of the sacks; one fell; how many have long hair, short hair; they are indoors etc. This means that in any situation there is a lot of potential data that can be collected, but we are not going to collect all – we need to focus on what it is that we need, what is important to monitor. We need to decide up front what it is that we need to know in order to measure performance and inform decisions. The caption of the picture was: “5 out of 6 children like sack-racing” This statement can be questioned, because it is based on an assumption that the child that fell does not like sack racing. This may not be true. Therefore, we need to think very carefully of our interpretation of the data. LEARNING POINT: Data collection can be simple, but be clear on what is data and what is interpretation.

6. Why do we collect data? To get feedback on how we are doing in relation to the results we want to achieve (e.g. performance monitoring) To know where we can improve our services (e.g. customer satisfaction) To find out how and where we can work more cost-effectively (e.g. cost-benefit analysis) To learn from our experiences and to inform our future planning (e.g. programme evaluation) What is it that we don’t we know, that if we know it, will make a difference if we know it?

7. TAU Context Building capacity of our clients to deliver services to citizens How is capacity defined? (individual, organisational, environmental) Focus on capacity in: OD, PPM, Strategic Planning, SCM and Finance Management etc. Capacity is built through process consulting and sometimes workshops or capacity building sessions What is our theory of change? (strategy, and logic model linking inputs>outputs>immediate, intermediate, ultimate outcomes – Results Based Management)

8. Let’s pause for a moment http://www.helentoons.com/2009/06/where-is-pause-button.html

9. Background on Results-Based Management • Before we look at data management on capacity building, we need to take note of how Results Based Management (RBM) works. • The next four slides (with recognition to Peter Brook – TAU) gives an overview of: • basic terminology; • the results chain; • levels of measurement on the logic model; • indicator definition; and • baselines, targets and measures

10. Activities, Outputs and Outcomes are the building blocks of Results Based Management Every programmeundertakesactivities that produceoutputs that contribute to the achievement ofoutcomes.

11. How do activities, outputs and outcomes link? Single Results Chain Results Hierarchy Outcome Output Activity Input

12. FINAL OUTCOME INTERMEDIATEOUTCOMES OUTPUTS ACTIVITIES All levels of the Logic Model should be measured Indicators Indicators IMMEDIATE OUTCOMES Indicators Indicators Indicators Indicators INPUTS

13. Indicator measurement has four different elements Indicator definition 1 • Identifies what should be measured, over what time period, and in what units, by who, how frequently, but does not contain a score or a target • The first measurement of an indicator Baselines 2 • For comparing later measurements with • For setting targets against • The score we would like to achieve Targets 3 • Must be time-bound, challenging and achievable • Actual score for each measurement 4 Measures • Must be measured regularly, so that trends can be established

14. Baselines, Targets and Measure are all expressed in the same units

15. So…now that we have refreshed our memories on RBM…what is the Programme logic/Theory of Change related to Capacity Development in TAU? http://www.iconspedia.com/icon/button-refresh-3374.html

16. Chain of events/levels of evidence

17. How do we know we are successful with capacity building?

18. How do we measure the indicators?

19. How do we collect the data? • Hard copy/electronic questionnaire • Telephonic, face-to-face or focus group interviews • When and how to administer questionnaires/use interview guides • Questionnaires: • How many questions? (Not too many, but cover the indicators) • Open-ended or fixed response categories • Options for fixed responses - even or uneven number, ranking scales, naming • The rhythm of the questionnaire

20. How do we capture and analyse and present the data?

21. …and we need to pause again…to consider a few technicalities… http://www.pvtmurphy.com/Prints/Pause%20Button.htm

22. What level of measurement/types of data are we working with? Nominal: Lowest, least precise level of measurement. Different types of categories of a variable (e.g. colour: red or blue or yellow/sex: male or female/religion: protestant/catholic). Categories are mutually exclusive, not ordered. Ordinal: Identifies difference among categories available, and allows categories to be rank ordered (e.g.: level of education: primary, secondary, tertiary) Interval: Categories are ranked, and distance between categories is measured. No true zero (e.g. IQ) Ratio: Most precise level, rank ordered, distance is precisely measured, and there is an absolute zero (e.g. age).

23. Please note - the level of measurement determines what calculations we can do with the data… The mode is the most frequent score. Can be use for all levels of measurement. (Most The median is the middle point – half of cases are above this point and the other half below. Can be used for ordinal, interval and ratio data. The mean is the average. Can be used only when we have interval and ratio data.

24. What you can and cannot do…

25. How can we present the data analysis? The most common ways of presenting analysed data are: • Pie charts • Bar graphs • Stacked bar graphs • Line charts

26. Pie Charts • Circles divided into a number of slices, each slice representing the relative proportion data points falling into a given category. • Suitable for graphing nominal data and percentages. http://www.mathleague.com/help/data/data.htm

27. Pie Charts Example: Attendance of TAU Breakfast Session 9 September 2010

28. Bar Graphs • Used to show umber of proportion of nominal or ordinal data according to a specific attribute (nominal data). • Often shows # or % of observations per data points • Switching rows and columns gives different perspectives (see next slide) –it is important to decide if you want to focus on the project phase or what the picture is per portfolio (see example in next slide) http://www.webdonuts.com/2009/08/the-presentation

29. Bar Graphs Projects per Project Phase (fictional data) Project phase per Portfolio (fictional data)

30. Stacked Bar Graphs Can be used for representing multiple categories of data in a single bar, and each bar is made up of sections that represents the proportion of data falling into a category. Can be used to show comparative contributions of different groups or ratings to a total. Use the switch rows and columns function to see which display makes sense – in the example in the next slide Participant Feedback (B) may be better to use.

31. Stacked Bar Graphs Participant Feedback (A) Participant Feedback (B)

32. Line Graphs A single line is used to connect data points (interval and nominal data) Used to present progress over time and trends More than one variable can be presented on a single chart – but check that it works. See examples on next slide: The presentation of # of website visits, together with page views per visit and # of seconds per page view does not work. The presentation of Male/Female breakdown over time works well.

33. Line Graphs • A single line is used to connect data points (interval and nominal data) • Used to present progress over time and trends http://www.netpaths.net/blog/

34. Line Chart Website report Staffing Table - Gender Breakdown per Quarter

35. Qualitative Data Qualitative data in questionnaires is mostly gathered through open-ended questions In the next slide a basic example is given of how qualitative data can be analysed: (1) Capture the text; (2) Group related themes/comments together; (3) count; (4) analyse; and (5) present the analysis in an appropriate format. NB Qualitative date provides more in-depth information, but if applicable, it can be counted, as in the example on the next slide

36. Qualitative data Future PMIG sessions should focus on…. Analysis Three priorities were identified for future PMIG sessions: Project office set-up and management (50%) Communication and stakeholder management (33.3%) Project Charter (16.7%) Verbatim • The project charter • How to set up a project office • Stakeholder management • Communication in projects • How to manage a Project office • Project Office

37. You can do it It is not as difficult as we sometimes think to analyse data, and to represent it in graphs There are, however a few points of caution and if you are not sure about something, rather consult a colleague that has a deeper understanding of data analysis.

38. It’s easy?

39. Really? Our potential President of the Math Club does not really qualify for this position – he has seriously under-estimated his support. Why? …see the next slide

40. Are you sure?

41. What happened here? This is a very common mistake Percentages cannot be added up as percentages A percentage represents a proportion of a universe, and the size of the universe needs to be taken into account. To calculate his support base correctly, our clever friend had calculate the total number of people that supported him in relation to the total number of boys an girls. LEARNING POINT: We need to know what calculations we can do with certain types of data. Always double-check. (see notes on types of data)

42. Thank You References: http://graphpad.com.faq.cfm?faq=1089 http://www.blueprintusability.com/topics/articlelevels.html http://highered.mcgraw-hill.com/sites/0073521426/student_view0/ebook16/chapter1/chbody1/levels_of_measurement.html http://www.socialresearchmethods.net/kb/measlevl.php http://courses.csusm.edu/soc201kb/levelofmeasurementrefresher.htm http://cnx.org/content/m10809/latest/ http://www.usablestats.com/lessons/noir http://www.thinkoutsidetheslide.com/articles/using_graphs_and_tables.htm http://www.mathleague.com/help/data/data.htm http://www.42explore.com/graphs.htm http://changingminds.org/explanations/research/measurement/types_data.htm