1 / 20

Objective Evaluation of Subjective Decisions

Objective Evaluation of Subjective Decisions. Mel Siegel & Huadong Wu Robotics Institute – School of Computer Science Carnegie Mellon University - Pittsburgh PA 15232 USA. SCIMA-2003 Soft Computing Techniques in Instrumentation, Measurement and Related Applications

carlos-wolf
Download Presentation

Objective Evaluation of Subjective Decisions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Objective Evaluation of Subjective Decisions Mel Siegel & Huadong WuRobotics Institute – School of Computer Science Carnegie Mellon University - Pittsburgh PA 15232 USA SCIMA-2003 Soft Computing Techniques in Instrumentation, Measurement and Related Applications Brigham Young University – Provo UT USA2003 May 17

  2. outline • background: problem of “sensor fusion for context aware computing” • approach: development of an “adaptive weighted Dempster-Shafer (D-S)” algorithm • issue (= the talk’s title): objective evaluation of subjective decisions • meta-issue: is it really an issue? • discussion: “receiver operating characteristic” • closing the loop: ROC  D-S ?

  3. background • “context detection” for HCI • e.g., your cell phone could ring louder if it could know it is in your briefcase • context detection requires subjective evaluation of “ordinary” sensor signals • sensor fusion required when we have multiple detectors, none of them very good • sequence of algorithms culminates in an “adaptively weighted Dempster-Shafer” method

  4. Camera View Focus-of-Attentiondecisionby fusion of video and audio data

  5. sensor fusion alternatives #1. complementary #3. cooperative Parametric template, Figures of merit, Syntactic pattern recognition … … Logical template AI rule-based reasoning, Heuristic inference Neural network … … #2. competitive

  6. our problem: Bayes can’t do it head pan left straight right sensor noise right observed pan straight left straight right right

  7. “Frame of Discernment” Θ lists all possibilities:{A}={ {L}, {S}, {R}, {L | S}, {S | R}, {L | R}, {L | S | R} } approach:the Dempster-Shafer method a theory of evidence allows belief and plausibility quantifies both knowledge and ignorance a generalization/extension of Bayesian inference network

  8. sensor fusion using “classical” Dempster-Shafer Theory of Evidence

  9. extension of Dempster-Shafer: evidence weighted by sensors’ reliabilities

  10. further extension of Dempster-Shafer: weights change according to performance history overcomes sensor drift problem!

  11. an arbitrary effectiveness measure

  12. generalizing via a simulation ... head pan left straight right sensor noise right observed pan straight left straight right right

  13. ... yields an intriguing resultwhen sensor precisions are very different

  14. the issue ... • objective evaluation of subjective decisions • a meta-issue: is it really an issue?

  15. “objective” vs. (?) “subjective” • in medicine the distinction is sharp: • subjective: means what the patient tells the physician about his/her complaint, what he/she thinks is the problem, etc • objective: means what the physician observes (and his/her instruments report) about the condition of the patient • statisticians talk about “rational gambling” • but in most contexts it feels fuzzier ...

  16. and even physicians make subjective decisions • whose quality we can evaluate objectively!:

  17. receiver operating characteristic • originally developed for target analysis • considers ratio of signal to signal-plus-noise vs. the discriminator level set • adopted and extensively developed in the medical diagnostic test community • { TP, TN }  signal, { FP, FN }  noise • most physicians understand a test’s sensitivity == TP/(TP+FN) andspecificity == TN/(TN+FP)vs. the chosen “cut point” of the test

  18. ROC • (dotted) ideal • (dashed) useless • reliable • (b) typical -- increasing cut point increases TPs (good) and FNs (bad) -- decreasing cut point increases TNs (good) and FPs (bad)

  19. 0 0 1 1 closing the loop? ... • ROC  D-S ? “plausibility” Dempster-Shafer “belief” evidence that supports X-- fever-- white tongue-- headache evidence that rules out X-- no virus detected-- had disease once before-- over age 55 TP TN ROC FN FP cut point

  20. conclusions / questions • adaptive weighted D-S seems to contribute an incremental but real improvement in appropriate sensor fusion applications • “objective”/“subjective” distinction is fuzzy • maybe ROC and related “cut point analysis” techniques can help us set neural net, fuzzy system, etc, parameters that are now set either arbitrarily or iteratively (hence slowly) • is the apparent connection between D-S and ROC superficial, or real at some deep level?

More Related