ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION: CONTRIBUTIONS FROM LAW

ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION: CONTRIBUTIONS FROM LAW DAVID A SCHUM, PROFESSOR SCHOOL OF INFORMATION TECHNOLOGY & ENGINEERING SCHOOL OF LAW GEORGE MASON UNVERSITY JON R. MORRIS, AFFILIATE PROFESSOR SCHOOL OF INFORMATION TECHNOLOGY & ENGINEERING GEORGE MASON UNVERSITY CARDOZO SCHOOL OF LAW JANUARY 20, 2007

SOME POINTS WE WILL ADDRESS IN OUR TALK • THE FIELD OF LAW HAS GIVEN ALL OF US A VERY RICH LEGACY OF EXPERIENCE AND SCHOLARSHIP REGARDING THE PROPERTIIES , USES AND DISCOVERY OF EVIDENCE. WE WISH TO APPLY SOME OF THIS LEGACY IN THE VERY DIFFICULT TASK OF ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION. • SOME CONCERNS ABOUT ATTRIBUTES OF THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES. • SOME THOUGHTS FROM EPISTEMOLOGY, LAW, AND COMMON EXPERIENCE SUGGEST WHAT COMPETENCE AND CREDIBILITY ATTRIBUTES OUGHT TO BE CONSIDERED. • MAJOR CONTRIBUTIONS FROM LAW: A LEGACY OF IMPORTANT QUESTIONS TO ASK ABOUT THE COMPETENCE AND CREDIBILITY ATTRIBUTES OF HUMAN WITNESSES. • PROBABILISTIC ISSUES IN THE ANALYSIS OF CHAINS OF REASONING IN WHICH CREDIBILITY CONSIDERATIONS FORM THE FOUNDATION FOR THESE CHAINS. • THESE VARIOUS THOUGHTS ARE COMBINED IN A COMPUTER-BASED SYSTEM WE HAVE BEEN WORKING ON FOR QUITE SOME TIME TO ASSIST OUR INTELLIGENCE ANALYSTS DO A MORE CAPABLE JOB OF ASSESSING SOURCES OF HUMINT [HUMAN INTELLIGENCE]. THIS SYSTEM IS CALLED "MACE" [METHOD FOR ASSESSING THE CREDIBILITY OF EVIDENCE]. TWO IMPORTANT USER COMMUNITIES FOR MACE: DEPARTMENT OF DEFENSE AND LAW ENFORCEMENT. 2

LAW AND INTELLIGENCE ANALYSIS • AMONG THE MOST VALUABLE CONTRIBUTIONS FROM LAW IS THE LEGACY OF EXPERIENCE AND SCHOLARHIP ON WITNESS COMPETENCE AND CREDIBILITY ASSESSMENT ACCUMULATED OVER THE PAST FIVE HUNRED YEARS OR SO IN OUR ANGLO-AMERICAN ADVERSARIAL LEGAL SYSTEM • WILLIAM TWINING ADVISES THAT WE SHOULD EMPHASIZE EXPERIENCE, SINCE MOST OF THESE INSIGHTS ARISE IN THE DAY TO DAY CRUCIBLE OF ADVERSARIAL EXPERIENCE IN TRIALS AT LAW. • WE HAVE BEEN ATTEMPTING TO EXPLOIT THIS LEGACY OF WISDOM IN OUR WORK FOR INTELLIGENCE ANALYSTS AND OTHERS WHO PROVIDE THE HUMAN INTELLIGENCE [HUMINT], ON WHICH SO MUCH DEPENDS. • OUR ANALYSTS ARE ASKED TO PERFORM CREDIBILTY ASSESSMENTS UNDER AN ALMOST UNBELIEVABLE ARRAY OF DIFFICULTIES AND NEED ALL THE HELP THEY CAN GET. • YOU WILL FIND NO BETTER ACCOUNT IN OPEN SOURCES OF THESE DIFFICULTIES THAN BY READING ABOUT OUR EXPERIENCE WITH "CURVEBALL" IN THE BOOK: ON THE BRINK, BY TYLER DRUMHELLER AND ELAINE MONAGHAN [CARROLL & GRAF, NY, 2006]. • IN THESE PERILOUS TIMES THE IMPORTANCE OF HUMINT CANNOT BE OVERSOLD. WHAT IS SO IMPORTANT ARE OUR EFFORTS TO DETERMINE WHETHER WE CAN BELIEVE WHAT HUMAN SOURCES TELL US. 3

FIRST CONTRIBUTION FROM LAW: DISTINGUISHING BETWEEN THE COMPETENCE AND THE CREDIBILITY OF HUMAN SOURCES ORTHOGONAL CHARACTERISTICS:ONE DOES NOT ENTAIL THE OTHER. EXAMPLE OF A FREQUENT MISTAKE: “We can believe what X tells us because he had good access to his sources” • COMPETENCE: • Appropriate sources, • In a position to observe, • Understanding of what was observed, • Ability to communicate. • CREDIBILITY: • Veracity, • Objectivity • Observational Sensitivity Arguments jusifying these attributes in a minute. THESE TWO CHARACTERISTIC ARE FREQUENTLY CONFUSED LEADING TO SERIOUS INFERENTIAL ERRORS. 4

A SEARCH FOR THE ATTRIBUTES OF THE CREDIBILITY OF HUMAN SOURCES OF EVIDENCE: THE “STANDARD ANALYSIS”OF KNOWLEDGE AND THE CHAIN OF REASONING IT SUGGESTS [controversial, but a good heuristic] A SOURCE “KNOWS” THAT EVENT E OCCURRED IF: E DID OCCUR, THE SOURCE GOT NON-DEFECTIVE EVIDENCE THAT E OCCURRED, AND THE SOURCE BELIEVED THE EVIDENCE THAT E OCCURRED. OUR INFERENCE ABOUT EVENT E, BASED ON THIS SOURCE’S TESTIMONY E* THIS SOURCE NOW TELLS US THAT E OCCURRED BASED ON AN OBSERVATION HE/SHE ALLEGEDLY MADE. WE HAVE UNCERTAINTY ABOUT THESE THREE EVENTS. HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR? DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE? DOES THIS SOURCE BELIEVE THAT E OCCURRED? THE SOURCE’S REPORT E* THAT EVENT E OCCURRED. OBSERVATIONAL SENSITIVITY OBJECTIVITY VERACITY 5

SOURCES OF SUPPORT FOR THIS CREDIBILITY-RELATED CHAIN OF REASONING: WIGMORE COMMON EXPERIENCE HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR? DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE? DOES THIS SOURCE BELIEVE THAT E OCCURRED? THE SOURCE’S REPORT E* THAT EVENT E OCCURRED. WAS THIS EVIDENCE ADEQUATE? HUMAN SENSE ARE FALLIBLE, ESPECIALLY UNDER CERTAIN CONDITIONS OBSERVATIONALSENSITIVITY PEOPLE DO NOT ALWAYS BELIEVE WHAT THEIR SENSES RECORD, BUT BELIEVE WHAT THEY EXPECT OR WISH TO HAVE OCCURRED DID THE WITNESS BASE A BELIEF ON SENSORY EVIDENCE/ OBJECTIVITY WITNESS DO NOT ALWAYS TESTIFY WHAT THEY BELIEVE PEOPLE DO NOT ALWAYS BELIEVE THE THINGS THEY TELL US VERACITY WITNESS TESTIMONY BASED ON PERSONAL OBSERVATION A PERSON TELLS US WHAT HE/SHE OBSERVED 6

A MAJOR CONTRIBUTION FROM LAW: GROUNDS FOR IMPEACHING OR SUPPORTING WITNESS CREDIBILITY VARIETY IS THE SPICE OF LIFE IN CREDIBILITY ASSESSMENT 7

THE MACE SYSTEM: MACE = METHOD FOR ASSESSING THE CREDIBILITY OF EVIDENCE MACE INITIALLY DESIGNED FOR CIA IN 1990; BUT THEY WERE NOT READY FOR IT. SINCE THE EVENTS OF 9/11/01 AND SUBSEQUENT EVENTS, THEY ARE MORE THAN READY FOR IT NOW. MACE CONSISTS OF TWO PARTS LEADING TO TWO DIFFERENT HEDGES ON A CONCLUSION ABOUT WHETHER TO BELIEVE WHAT A HUMAN SOURCE TELLS US. • PART II. • HOW STRONG IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BAYESIAN QUESTION]. MACE ALLOWS THE USER TO: • ASSESS LIKELIHOODS ASSOCIATED WITH THE THREE CREDIBILITY ATTRIBUTES, BASED ON EVIDENCE MARSHALED IN PART I. • COMBINES THESE LIKELIHOOD ASSESSMENTS USING THE ALGORITHM JUST GIVEN. • CALCULATES POSTERIOR ODDS ON WHETHER WE SHOULD BELIEVE WHAT THIS SOURCE HAS TOLD US. • PART I. • HOW COMPLETE IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BACONIAN QUESTION]. MACE ALLOWS THE USER TO: • MARSHAL ANSWERS TO THE COMPETENCE AND CREDIBILITY QUESTIONS SUGGESTED BY 500 YEARS OF EXPERIENCE IN LAW, • JUDGE WHETHER THE ANSWERS FAVOR OR DISFAVOR THE SOURCE’S COMPETENCE AND ATTRIBUTES OF HIS/HER CREDIBILITY, • DECIDE WHETHER THE EVIDENCE JUSTIFIES A BELIEF IN WHAT THE SOURCE HAS REPORTED [THIS PART MIGHT STAND ON ITS OWN]. 9

PART I IN MACE ALLOWS THE MARSHALING OF ANSWERS TO THESE COMPETENCE AND CREDIBILITY QUESTIONS COMING FROM LAW VERACITY 1 PRIOR INCONSISTENCIES 2. OUTSIDE INFLUENCES 3. EXPLOITATION POTENTIAL 4. CONTRADICTION AND CONFLICT 5. CORROBORATION AND CONFIRMING 6. CHARACTER 7.REPORTING RECORD 8. TAILORING INFORMATION 9. COLLATERAL DETAILS 10. INTERVIEW BEHAVIOR COMPETENCE 1. ACCESS TO INFORMATION 2. KNOWLEDGE AND EXPERTISE. 3. OBSERVATIONAL CAPABILITIES. 4. MEETING BEHAVIOR 5. MOTIVATIONAL CONSISTENCY 6. RESPONSIVENESS. SENSITIVITY 1. SENSORY CAPACITY 2. OBSERVATIONAL CONTEXT 3. PAST ACCURACY 4. CONTRADICTION AND CONFLICT 5. COLLATERAL DETAILS OBJECTIVITY 1.OBSERVATIONAL EXPECTATIONS 2.OBSERVATIONAL DESIRES 3. BELIEF CONSEQUENCES 4. MEMORY EFFECTS 5. CONTRADICTION AND CONFLICT USERS RESPOND TO ANSWERS TO THESE QUESTIONS BY CONSIDERING THE EVIDENCE AT HAND FOR THE SOURCE AND CHECKING ONE OF THE FOLLOWING BOXES: ANSWER IS FAVORABLE TO AN ATTRIBUTE: I CAN’T DECIDE: ANSWER IS DISFAVORABLE TO AN ATTRIBUTE: NO ANSWERS TO THIS QUESTION: 10

OVERALL BACONIAN ASSESSMENT IN PART I OF MACE YOUR OVERALL JUDGMENT ABOUT THIS REPORT: INTELLIGENCE SUPPORTS THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR: INTELLIGENCE FAILS TO SUPPORT THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR: CAN’T DECIDE: JUDGEMENT BASED ON THIS SUMMARY:AN EXAMPLE ANSWERED QUESTIONS UNANSWERED QUESTIONS 2 1 2 COMPETENCE [5 Qs] VERACITY [10 Qs]: OBJECTIVITY [5 Qs]:\ SENSITIVITY [5 Qs}: 2 5 3 3 1 1 2 1 2 8 7 3 2 CREDIBILITY SUMMARY: 11

MACE PART II. BAYESIAN ASSESSMENTS OF THE STRENGTH OF EVIDENCE REGARDING THE CREDIBILITY OF A HUMINT SOURCE. ASSESSMENTS OF PAIRS OF LIKELIHOODS FOR EACH CREDIBILITY ATTRIBUTE: 1.0 1.0 1.0 P(E*|EB) P(ES|E) P(EB|ES) 0 0 0 1.0 P(EB|ESC) 1.0 P(E*|EBC) 1.0 P(ES|EC) FOR VERACITY FOR OBJECTIVITY FOR SENSITIVITY OR ACCURACY • ASSESSMENTS BASED ON ALL EVIDENCE FOR EACH ATTRIBUTE • ASSESSMENTS CAN BE MADE IN GRAPHICAL FORM AS SHOWN ON THE NEXT SLIDE. 12

AN EXAMPLE OF LIKELIHOOD ASSESSMENT: FOR VERACITY ANALYSTS, AS WELL AS OTHERS, RESIST HAVING TO MAKE NUMERICAL ASSESSMENTS OF PROBABILITIES. SO, MACE ALLOWS THEM TO DRAW BOXES IN A TWO DIMENSIONAL PROBABLITY SPACE [ONE BOX PER ATTRIBUTE]. 1.0 LOCATION OF BOX TELLS HOW STRONGLY THE EVIDENCE FAVORS OR DISFAVORS VERACITY. P(E*|EB) SIZE OF BOX SHOWS USER CONFIDENCE IN HIS/HER ASSESSMENT 0 1.0 P(E*|EBC) 13

AUTOMATIC CALCULATIONS MADE BY THE MACE SYSTEM FIRST: MACE FINDS THE FOUR NUMERICAL CORNER POINTS OF EACH OF THE THREE BOXES DRAWN BY THE USER; ONE BOX PER ATTRIBUTE. SECOND: MACE FORMS ALL 64 COMBINATIONS OF THESE CORNER POINTS. THIRD, FOR EACH COMBINATION, MACE CALCULATES h = P(E*|E) AND f = P(E*|EC) USING THE ALGORITHM PREVIOUSLY GIVEN: For h = P(E*|E) = P(ES|E)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EBC)] + P(EB|ESC)[P(E*|EB) - P(E*|EBC)] + P(E*|EBC). For f = P(E*|EC) = P(ES|EC)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EBC)] + P(EB|ESC)[ P(E*|EB) - P(E*|EBC)] +P(E*|EBC). FORTH, MACE FINDS ALL POSSIBLE VALUES OF THE 64 POSSIBLE h/f = P(E*|E}/P(E*|E) RATIOS AND FINDS AN INTERVAL CONTAINING ALL OF THEM. NOTE:THE RATIO h/f = P(E*|E}/P(E*|E) SHOWS THE BAYESIAN FORCE OF EVIDENCE IN AN INFERENCE ABOUT WHETHER E IS TRUE, BASED ON THE SOURCE’S REPORT E*. 14

FINALLY, MACE PROVIDES THE FOLLOWING GRAPHICAL DISPLAY: A SINGLE EXAMPLE PRIOR ODDS INTERVAL. GIVEN INITIALLY BY USER LIKELIHOOD RATIO INTERVAL CALCULATED BY MACE POSTERIOR ODDS ON E , GIVEN THE SOURCE’S REPORT. __ FAVORS EC 0 + FAVORS E A LOG ODDS OR LOG LIKELIHOOD RATIO SCALE 15

WHAT'S NEXT IN THE DEVELOPMENT OF MACE? • EXTENDING MACE TO DEAL WITH ADDITIONAL CREDIBILITY MATTERS WHEN HUMAN SOURCES PROVIDE TANGIBLE EVIDENCE. • EXTENDING MACE TO COPE WITH “WILDERNESS OF MIRRORS” PROBLEM, WHEN WE HAVE CHAINS OF SOURCES ALL COMMENTING ON EACH OTHER’S CREDIBILITY. • ENHANCING MACE TO ALLOW BETTER RECOGNITION OF DECEPTIVE HUMAN BEHAVIOR. MANY THANKS FOR ALLOWING US TO SHARE WITH YOU PARTS OF THE STORY OF MACE. THAT'S ALL FOR NOW! 16

ASSESSING THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES OF INTELLIGENCE INFORMATION: CONTRIBUTIONS FROM LAW