1 / 9

Comparable Evidence: subjects, objects and context?

Comparable Evidence: subjects, objects and context?. Claes Wohlin Blekinge Institute of Technology Sweden. Session outline. Introduction: Comparable Evidence across Experiments, Claes Wohlin Readers Project: Quality Aspects in Empirical Studies, José Maldonado

venus-holman
Download Presentation

Comparable Evidence: subjects, objects and context?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comparable Evidence:subjects, objects and context? Claes WohlinBlekinge Institute of TechnologySweden

  2. Session outline • Introduction: Comparable Evidence across Experiments, Claes Wohlin • Readers Project: Quality Aspects in Empirical Studies, José Maldonado • Witch Doctors and Sugar Pills, Barbara Kitchenham Presentation specific discussions after the two latter presentations and then a general discussion at the end.

  3. Evidence? Individual studies produce: • Evidence Admissible evidence • Circumstantial • evidence • Can we make a case? • Sufficient evidence? • Trustworthy evidence? Screening What makes evidence admissible?

  4. Why this session? Concerns when answering the two questions! • What makes evidence admissible? • Can we make a case? Experimentation in software engineering: We are trying to use experiments as a means for change, for example as part of technology transfer. This means that evidence from experimental studies must be comparable, but not too similar.

  5. Subjects • What is the sample? • What is the population? • Are the subjects sufficiently motivated? • Are the subjects capable of performing the task as would be expected by the intended population? • What does it for example mean to assign roles in PBR in comparison to actually using subjects having the roles?

  6. Objects • How do we measure that objects are of the same difficulty for the subjects? • Do any of the treatments require more skills and experience than the other? • Are the treatments (methods/techniques/tools) described to the same level of detail? • Is the education into the treatments fair (on the same level)?

  7. Context • Does the context of the objects resemble the true use? • What does it mean to run experiments off-line? • How do we create a sufficiently realistic context?

  8. Research • How do we ensure that all of the above is similar enough when replicating experiments? • How much replication? By whom? • Are lab packages a good idea? Too similar experiments? • How do we ensure that different studies do not intentionally or unintentionally favor one of the treatments over the other? Especially inventor evaluate her/his method against something else.

  9. Presentations • José Maldonado • Barbara Kitchenham We will hopefully have some time to return to the questions above at the end of the session.

More Related