Shake, Rattle and Roles: Earthquake Engineering as HRO

1. Shake, Rattle and Roles: Earthquake Engineering as HRO Dan Horn Jeremy Birnholtz November 5, 2003

2. Presentation Outline • Introduction • High Reliability Organizations • Earthquake Engineering • Methods • Findings • Implications • Next steps

3. High Reliability Organizations • Weick (1999) lists 3 key characteristics • Environment rife with errors to be detected • Constant monitoring and reporting • No anomaly too small • Reluctant to simplify interpretations • Integrate multiple, redundant sources • Ongoing sensitivity to operations • Collective mindfulness, heedful interrelating, the “bubble” etc.

4. Why study HROs here? • Error detection is a key user goal, and therefore a key aspect of design • Lessons are broadly applicable • Weick suggests that ordinary orgs become HRO-like in crises • Newell (1997) suggests that under extraordinary circumstances, the ordinary user becomes extraordinary

5. Experimental Structural Earthquake Engineering (EE) • Large scale physical test equipment • Many forces that are complex and interacting • Potential danger!

6. Structural Labs as HROs • We argue that structural EE labs are a form of HRO • 3 types of risk faced in these labs • Catastrophic specimen failure • Losing laboratory and field autonomy; as (Galison, 1997) discusses in physics • Risk of significant social cost; concentrated social risk (Sims, 1999)

7. NEES: Telepresence • Teleobservation • Watching tests from a distance • Potentially many observers • Passive vs. active • Teleoperation • Controlling apparatus during test • Remote operations

8. Methods • Interviews with 75 earthquake engineers • Faculty, students and technicians • Questions about • Sequence of a typical test • What they are looking at during a test • Ongoing observation of EE tests • Coding for themes (Huberman and Miles, 1994)

9. Findings • Local Failure detection • Variable Likelihood of failure • Integrating sensory cues • Multiple collocated persons • Beliefs about telepresence • Remote Failure Detection: One story

10. Variable likelihood of failure • More likely early and late: • “I’m always there for the first test on a particular specimen, because I need to train the students on the things they need to do … like making sure the test frame is not creating a physical anomaly. Students have a tendency to just roll forward without checking these things.” • Early failure • Dangerous and Costly • Late failure • Predictable and can be prepared for this

11. Integration of sensory cues • Students rely on visual cues • Visual displays of data (e.g. graphs) • Walking around and looking at the specimen • “if we can’t explain the graphs, we stop immediately. If we get data that are surprising, but not crazy we’ll keep going” • More experienced integrate more cues • Sound: “Even after [we had fixed a problem with the test setup], there was still a lot of noise. I might have pushed the [emergency stop] button, as it was very noisy.” • “Feel”

12. Multiple Collocated Persons • Reliance on multiple viewpoints • “different accounts of what happened, like people’s reports at the scene of a car accident“ • Technicians say they will “send somebody out to stand in a particular place and keep an eye on things.” • Frequent interaction • “When things go awry, we tend to powwow in the lab…to sort out what’s going on.” • “You have to argue”

13. Beliefs About Telepresence • Value in remote observation • Dog-and-pony show value • “Real” researchers plan to be at their tests • Fears of remote operation • Don’t mess with my actuators • Low-fidelity means low value

14. Remote Participation: One Story • One faculty member had remote participants in his shake table test when he was a graduate student. He had primitive video via html and people watching could email him. • Valuable in that • “you’re concentrating on one thing like maybe running the test, and someone emails you and says, ‘hey what’s that that’s going on?’ and you look right there and you get a whole other opinion about what’s going on” • Email is low-cost, persistent and not real-time: “Don’t have 20 people yelling at you at once.”

15. Implications • Additional local observers are valuable • But: tend to have more correlated views • Remote observers are constrained • Cues are less rich • Cues are mediated • Can we design representations that exploit these constraints? • Increase statistical probability of error detection

16. Formal Model P(D|F)=1-(1-P(d1|F))…(1-P(dn|F)) Person 1: .5 Person 2: .4 P(D|F)=1-(.5)*(.6)=.7 = 70% ASSUMES STATISTICAL INDEPENDENCE

17. Next step: Lab Studies • Background • Task • Expected results

18. Experiment: Background • Tower of Hanoi (TOH) Problem • Rule 1: Only move 1 piece at a time • Rule 2: Only move smallest FROM a peg • Rule 3: Only place smallest ONTO a peg

19. Experiment: Background • Waitress and Orange TOH Isomorph (Zhang & Norman, 1994) • Converts part of external representation into internal representation • Leads to less efficient performance

21. Experiment: Background • Distributed Representations in TOH (Zhang, 1998) • Waitress and Orange Problem • Different Levels of Knowledge • Expert: R123 • Mid-level: R12 or R13 • Novice: R1 • Pairs – Taking Turns • R123-R1 vs R12-R13 vs R123-R123 vs R123

22. Zhang’s Results: Solution Time

23. Zhang’s Results: Steps

24. Zhang’s Results: Errors

25. Zhang’s Results: Summary • Two experts are always better than one • Two mid-levels are never better than one expert • A novice is as helpful as a second expert in reducing errors

26. Beyond Zhang: Proposed Study • Payoff (to be piloted) • +$1.00 per solved problem • -$0.10 per move • -$0.40 per error • “Local” person • Knows all rules • Complex display, full information • Makes all moves

27. Beyond Zhang: Proposed Study • “Remote” person(s) • Knows all rules • Complex vs. Simple display (full vs. partial info) • Can only reject moves • Illegal • Inefficient • Either illegal or inefficient • Rejections cost $0.20 each

28. Proposed Study • Expected Outcomes (Remote Interface) • Simple > Complex in Error Detection • Simple < Complex in Inefficiency Detection

29. Implications • How these lessons will inform design of these systems • Could show value for remote folks • Allow us to take this to EE context • New expert-remote participants • Must overcome “speedbump of inefficiency”

30. Questions?