1 / 64

Masked Visual Analysis (MVA)

Masked Visual Analysis (MVA). A method that allows visual analysts to guard against falsely concluding an intervention has an effect. What should I do?. MVA Steps for Response-Guided Randomized Designs. 1. Set study parameters Research team agrees upon: Deign type (e.g., MB)

abadie
Download Presentation

Masked Visual Analysis (MVA)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Masked Visual Analysis (MVA) A method that allows visual analysts to guard against falsely concluding an intervention has an effect

  2. What should I do?

  3. MVA Steps for Response-Guided Randomized Designs 1. Set study parameters Research team agrees upon: • Deign type (e.g., MB) • Minimums (e.g., minimum of 5 observations per phase) • Randomization (e.g., random order of participants in MB)

  4. 2. Split into two teams Analysis Team Visually analyze the data and direct the Intervention Team Intervention Team Conduct the study based on the agreed upon parameters and the direction of the Analysis Team

  5. 3. Conduct the study • The Intervention Team begins the study and sends the collected outcome data to the Analysis Team • The Analysis Team analyzes the data and makes decisions about when it would be appropriate to make a random assignment • The Intervention Team makes random assignments when directed by the Analysis Team and continues to collect and send the outcome measures to the Analysis Team, but they never disclose the results of the random assignments • The Analysis Team indicates when the study should be concluded

  6. 4. Compute the p-value • The Analysis Team specifies what they believe are the results of the random assignments • The Intervention Team indicates if they are correct • If not correct, the Analysis Team continues to make specifications until a correct specification is made • The p-value is computed as: p = # specifications/# possible assignments

  7. Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters Dependent Variable? % of time on task Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 2 observations Treatment phases with at least 3 observations

  8. Example 1: Multiple Baseline Design – 4 Participants Step 1: Set study parameters Randomization? Randomize order of participants for intervention How many possible assignments of participants to treatment order? Who is 1st, 2nd, 3rd, and 4th? P = 4! = 24 possible assignments If the treatment has no effect, the probability that a masked visual analyst could identify the correct order p = 1/24 = .0417

  9. Example 1: Multiple Baseline Design – 4 Participants Step 2: Split into two teams Step 3: Conduct the study

  10. Wendy % Time on Task Tom Session

  11. Wendy Rob Tom Joel

  12. Wendy Rob % Time on Task Tom Joel Session

  13. Wendy Rob % Time on Task Tom Joel Session

  14. Wendy Rob % Time on Task Tom Joel Session

  15. Wendy Rob Tom Joel

  16. Wendy Rob Tom Joel

  17. Wendy Rob Tom Joel

  18. Wendy Rob Tom Joel

  19. Example 1: Multiple Baseline Design – 4 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the correct order p = 1/24 = .0417

  20. Example 2: Multiple Baseline Design – 3 Participants Step 1: Set study parameters Design Type? Multiple Baseline Across Participants Minimums? At least 5 baseline observations Staggers of at least 3 observations Treatment phases with at least 5 observations If outlier, at least 3 additional observations in phase

  21. Example 2: Multiple Baseline Design – 3 Participants Dependent Variable? % intervals with prosocial behavior Randomization? - How many possible assignments of participants to treatment order? Who is 1st, 2nd, and 3rd? P = 3! = 6 possible assignments - What if we randomly select from Participant 1, Participant 2, Participant 3, and no one? P=4! = 24 possible assignments, if correct p = 1/24 = .0417

  22. Example 2: Multiple Baseline Design – 3 Participants Step 2: Split into two teams Step 3: Conduct the study

  23. Ann James John

  24. Ann James John

  25. Ann James John

  26. Ann James John

  27. Ann James John

  28. Ann James John

  29. Ann James John

  30. Example 2: Multiple Baseline Design – 3 Participants Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? If the treatment has no effect, the probability that a masked visual analysts could have identified the assignments p = 1/24 = .0417

  31. Example 3: Changing Criterion Design Step 1: Set study parameters Design Type? Changing Criterion Minimums? At least 3 observations per phase Randomization? Random select without replacement from {M,M,M,M,M,M,S,S} for each phase to determine whether to Move to the next level of the criterion (M) or stay at the current level (S) Number possible assignment is 28 So if correct on first specification, p = 1/28 = .0357

  32. Example 3: Changing Criterion Design Step 2: Split into two teams Step 3: Conduct the study

  33. Example 3: Changing Criterion Design Baseline ? ? ? ? ? ?

  34. Example 3: Changing Criterion Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/28 = .0357

  35. Example 4: Alternating Treatments Design Step 1: Set study parameters Design Type? Alternating Treatments (2 treatments) Minimums? At least 5 alternating pairs Randomization? Random assignment of one observation in the pair to A and one to B Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05. 25=32, so if correct with 5 pairs, p = 1/32 = .03125

  36. Example 4: Alternating Treatments Design Step 2: Split into two teams Step 3: Conduct the study

  37. Example 4: Alternating Treatments Design Step 4: Compute the p-value Analysis Team make a specification Intervention Team, are they correct? Yes? p = 1/64 = .015625 No? Make a second specification If correct this time, p = 2/64 = .03125

  38. Example 5: Reversal Design • Step 1: Set study parameters • Dependent Variable? Number of Disruptive Behaviors • Design Type? Reversal • Minimums? At least 5 observations per phase • At least 3 phase changes (at least ABAB) • Randomization? Random assignment of treatment to blocks of • observations • Because each assignment has 2 possibilities, need 5 assignments to obtain over 20 possible assignments and a p-value < .05. • 25=32, so if correct p = 1/32 = .03125

  39. Example 5: Reversal Design Step 2: Split into two teams Step 3: Conduct the study

More Related