Themis IA testing: Lessons learnt from performing study

1. Themis IA testing:Lessons learnt from performing study Scott Rippon, User Experience ConsultantITS > Web Services > User Experience Design team

2. Table of contents • IA testing background • Lessons: • L1: Creating Confidence Interval (CI) for discrete data • L2: Cleaning continuous data & calculating mean and CI • L3: Free text fields are evil! • L4: Adding straight line to column chart • L5: Analysing for particular audience groups was manual & time consuming

3. IA testing background • Themis is the UoM HR system. • IA testing: • Goal: evaluate the usability of the Information Architecture (IA) of the present Themis system. Does the grouping/labeling make it easy for staff to find content? • Use OptimalWorkshop’sTreeJack tool to perform study online. • Performed in Dec 2012.

4. L1: Creating Confidence Interval (CI) for discrete data • Definitions: • Discrete data = Data that can only take certain values (eg. task success: Pass = 1; Fail = 0). • Confidence interval = Provides a value that we can ± to our mean/average. If we repeat the study with a different sample we can be X% certain their results will fall between this range. • Essential reading:Albert, W., Tullis, T., Tedesco, D. (2010), Beyond the Usability Lab, Morgan Kaufmann. • Used formula from Calculating a Confidence Interval for Task Completion (Measuring the UX)

5. L1: Creating Confidence Interval (CI) for discrete data cont. • Used CI to add error bars to graph. • Steps: • Right clicked Scores bar, selected ‘Format Data Series…’ • Select ‘Error Bars’ from LHS menu. • In ‘Display’ group select ‘Both’. • In ‘Error amount’ group click [Specify Value] button. • Select CI values from CI column for both the ‘Positive (& Negative) Error Value’ fields. * For Mac Excel 2008.

6. L2: Cleaning continuous data & calculating mean and CI • Definition: • Continuous data = Data that can take any value within a range (eg. time, height, 1 to 7 scores). • Pre-test questionnaire contained number of questions asking participants to rate whether they agreed (1 to 7). • Excel cells contained same text as form (eg. ‘1 - Strongly disagree’). Cannot use this label to calculate mean/average. • Solution: • Created new column for calculation. • Multiplied rating (C57) by responses (E57). See below figure. • Calculated average by dividing total of these multiplication (SUM(F57:F63))by total responses (SUM(C57:C63)). • See L1 for creating & graphing CI.

7. L3: Free text fields are evil! • Study had 2 free text fields: • Pre-test questionnaire: training completed • Had to manually recode all the responses. • Time consuming and had to make a number of assumptions. • Should have present participants with set list. • Post-test questionnaire: other feedback • Created individual cards and performed card sort (AKA Affinity Diagramming) exercise to group responses and find themes in the data.

8. L4: Adding straight line to column chart • Added success goal benchmark to graph (Guy’s idea). • Hair pulling experience!

9. L4: Adding straight line to column chart cont. • Steps: • Created new column with 75%. • Highlighted scores and goal columns & created column graph:‘Charts’ tab, [Column] button, [Clustered Column] button • In the graph clicked the ‘Goal’ bars then changed it to a line graph:‘Charts’ tab, [Line] button, [Line] button * For Mac Excel 2008.

10. L5: Analysing for particular audience groups was manual & time consuming • TreeJack doesn’t allow you to filter the results for a particular audience group. • To calculate scores for a particular audience group had to be done manually by manipulating Excel. • Pain!!!

11. Thank you… • If anyone would like copy of the Excel spreadsheet please contact me...Scott.Rippon@unimelb.edu.au