The Nemesysco Scandal – A Report

1. ICSI The Nemesysco Scandal – A Report John J Ohala Professor Emeritus, Linguistics, UCB

2. Outline 1. Nemesysco, their beginnings, their products, and how they promote them. 2. The Eriksson & Lacerda paper re Nemesysco 3. Press coverage of Nemesysco & of their threats to the publisher of the Eriksson & Lacerda paper. 4. Coverage in the ‘Blogosphere’. 5. Who is the guy behind all this? 6. What can be done?

3. 1. Nemesysco, their beginnings, their products and how they promote them.

4. Starting around 1997 or 2000 (its not clear when) an Israeli firm, Nemesysco, started promoting and selling devices that were said to be able to detect emotion and stress in a speaker via an analysis of their voice. It used an analysis techmique called ‘layered voice analysis’ or LVA for short. Their claim : “Nemesysco’s Layered Voice Analysis (LVA) technology detects and measures the emotional content of human speech, captured live or extracted from recorded audio. LVA identifies various types of stress, cognitive processes and emotional reactions which together comprise the �emotional signature� of an individual at a given moment, based solely on the properties of his or her voice . The technology detects minute, involuntary changes in the voice reflective of various types of brain activity. By utilizing a wide range spectrum analysis to detect minute changes in the speech waveform, LVA detects anomalies in brain activity and classifies them in terms of stress, excitement, deception, and varying emotional states, accordingly. This way, LVA detects what we call �brain activity traces,� using the voice as a medium. The information that is gathered is then processed and analyzed to reveal the speaker�s current state of mind. Segue to their professionally desgined website (live).

5. In their promotions they emphasize their technique is ‘patented’. Everyone should know that it is not that difficult to get something patented; all you need to do is follow the format dictated by the patenting agencies. (Aside: even I have a patent; look it up!) As far as I can tell, there are two US patents awarded to Amir Liberman. There are other patents filed in other countries. This one will become important later on in this presentation.

6. Further elaboration of their claims: LVA Analysis Process LVA has two basic formulas comprised of unique signal processing algorithms that extract more than 120 emotional parameters from each voice segment. These are further classified into nine major categories of basic emotions. Depending on the goal of the analysis, up to eight “final analysis” formulas can be applied to the emotional parameter data. These include: Lie stress analysis, Arousal level, Attention level, Emotional level, Conflict level, Deception patterns match, and additional methods for veracity assessment Just to note one of many inconsistencies: Elsewhere on their website they insist that their products are NOT ‘lie detectors’.

7. Window dressing to make the customrer think they are experts in speech. The Human Speaking Mechanism The human speaking mechanism is one of the most complicated procedures the human body is capable of, due to the number of muscles and physical apparatus involved, and the ways in which they need to be synchronized in perfect timing.   Initially, the brain apprehends a given situation and the possible implications of whatever will be said. Then when a person decides to speak, air is pushed upward from the lungs into the vocal cords. This causes the vocal cords to vibrate at a specific frequency and produce sound. The vibrated air continues to flow up toward the mouth where it is manipulated by the tongue, teeth and lips to produce sound streams which we interpret as words or phrases.  The brain closely monitors all of these procedures, ensuring that the sound emitted is what was intended, is intelligible, and is at a volume that can be heard by the intended listener. Due to this constant cerebral monitoring, every "event" that passes through the brain will leave a trace on the speech flow. LVA technology ignores what your subject is saying (i.e., the specific content) and focuses only on changes in brain activity that are reflected in the voice. In other words, what is critical is not “what” your subject is saying, but “how” he or she says it.

8. Nemesysco has spiffy computer interfaces:

9. A Figure from the patent of 17 Jan 2007.

10. To those unschooled in the basics of speech communication, Nemesysco describes their methods in language that is bound to impress them: it sounds scientific and authoritative. SENSE Analysis Process The SENSE technology is comprise of 4 sub-processes: 1. The vocal waveform is analyzed to measure the presence of local micro-high frequencies, low frequencies and changes in their presence within a single voice sample. 2. A precise frequency spectrum of the vocal input is sampled and analyzed. 3. The parameters gathered by the previous steps are used to create a baseline profile for the subject. 4. The new voice segments to be tested are compared with the subject's baseline profile, and the analysis is generated.

11. Nemesysco cites a number of “studies” supposedly giving evidence of the efficacy of their products. As documented in the blogs about Nemesysco, a number of these studies are worthless, being conference papers or posters at conferences where there was no reviewing process, or where the authors have some connection to or financial interest in promoting Nemesysco’s products. But let’s examine one study where the above discounting factors are not involved.

12. One document cited on the Nemsysco website in support of one of their early products called ‘Vericator’ (one thing Nemesysco does well is coining clever names for itself and its products) is said to report that "A three-year study by Air Force Research Laboratory (AFRL) Information Directorate engineers has concluded that several features of voice stress analysis are effective for detecting when a person is answering questions under stress..." The report: Investigation and Evaluation of Voice Stress Analysis Technology Final Report February 13,2002 Darren Haddad, Sharon Walter AFRUIFEC Rome Research Site Roy Ratley Megan Smith ACS Defense Rome, NY

13. Parsing this study: 1. This (altered) quote is based on anecdotal evidence from one law enforcement officer, not a proper controlled test. 2. The whole report does not include results from any other properly controlled test. 3. Moreover, the report contains factual errors.

14. This statement, in fact, is false. Shipp & Izdebski (JASA 1981) studied this & found no such microtremor in the laryngeal muscles (but did find it in limb muscles; moreover the micro-tremor is found when the muscle is resting; not when it is active. It contains the statement: “VSA literature [9] points to a descriptor of the physiological basis for the micro muscle tremor or microtremor. This paper describes "a slight oscillation at approximately 10cycles per second" (i.e. physiological tremors) during the normal contraction of voluntary muscle. All muscles in the body, including the vocal chords [sic], vibrate in the 8 to 12 Hz range. It is these microtremors that the VSA vendors claim to be the sole source of detecting if an individual is lying. “

15. And, one might ask, why cite this study when on their own website, Nemesysco declares: “Layered Voice Analysis [LVA} is not a "Voice Stress" analysis technology, nor does it use any previously known method for detecting voice stress. As such, LVA does not perform "micro-tremors" analysis in the voice.”

16. Further suspicious omission: In reviewing the literature on VSA, this very same report mentions: The Nemesysco product evaluated.

17. My own take on this ‘study’: As for the officer quoted in the report, Michael G. Adsit, Criminal Investigator, and the authors of the report to the US Dept of Justice, Darren Haddad, Sharon Walter, Roy Ratley, & Megan Smith, none of them really know much about the motor control of the speech apparatus. They don’t control the vast literature in this area. (They may have expertise in other domains: speech processing, speech compression.)

18. Finally, the Nemesysco list of ‘studies’ neglects to mention two studies that did attempt to implement proper controls: 1. Hollien, Harry; James D. Harnsberger (2006-03-17), "Voice Stress Analyzer Instrumentation Evaluation" (pdf), CIFA Contract – FA 4814-04-0011, http://www.clas.ufl.edu/users/jharns/Research%20Projects/UF_Report_03_17_2006.pdf] “device showed significant sensitivity to the presence of stress or deception in the speech samples tested. The true positive and false positive rates were parallel to a great extent “ 2. Damphousse, Kelly R. (March 2008). "Voice Stress Analysis: Only 15 Percent of Lies About Drug Use Detected in Field Test". NIJ Journal (National Institute of Justice) “Does VSA work? As our findings revealed, the two VSA programs that we tested had approximately a 50-percent accuracy rate in detecting deception about drug use in a field (i.e., jail) environment; however, the mere presence of a VSA program during an interrogation may deter a respondent from answering falsely.”.

19. 2. The Eriksson & Lacerda paper re Nemesysco (& one other similar product named “Diogenes”)

20. Then came the paper by Anders Eriksson and Francisco Lacerda: “Charlatanry in forensic speech science: A problem to be taken seriously” The International Journal of Speech, Language and the Law (Publisher: Equinox), IJSLL vol 14.2 2007 169–193.” Papers published in IJSLL are peer-reviewed.

21. Francisco Lacerda, Professor of Phonetics at Stockholm University Anders Eriksson Professor of Phonetics Dept. of Philosophy, Linguistics and Theory of Science University of Gothenburg

22. Eriksson, A. & Lacerda, F. (2007)"Charlantry in forensic speech science: A problem to be taken seriously“Intl J. Speech Lang. Law 14, 169– 173. ABSTRACT A lie detector which can reveal lie and deception in some automatic and perfectly reliable way is an old idea we have often met with in science fiction books and comic strips.  This is all very well.  It is when machines claimed to be lie detectors appear in the context of criminal investigations or security applications that we need to be concerned.  In the present paper we will describe two types of "deception" or "stress detectors" (euphemisms to refer to what quite clearly is known as "lie detectors")." Both types of detection are claimed to be based on voice analysis but we found no scientific evidence to support the manufacturers’ claims.  Indeed, our review of scientific studies will show that these machines perform at chance level when tested for reliability." Given such results and the absence of scientific support for the underlying principles it is justified to view the use of these machines as charlatanry and we argue that there are serious ethical and security reasons to demand that responsible authorities and institutions should not get involved in such practices.

23. Their paper reviewed existing literature but did not report the results of a controlled study, unlike those of Hollien & Harnsberger (2006) or Damphousse(2008). Rather, as concerned the Nemesysco product, they they did something more fundamental and ultimately more important – to inform the knowledgeable speech tech community: they examined the publically accessible patent for the device. What they found: “[To say that] there is absolutely no scientific basis for the claims made by the LVA proponents is an understatement. The ideas on which the products are based are simply complete nonsense.” They also offer what we may characterize as a “sociological” account of Nemesysco’s economic success: “While, as we have seen, the voice stress detectors are not of any real use as the lie or stress detectors they are claimed to be, they have certainly not been without success in other areas. One such area is making money for the vendors.”

24. The statistics are based upon what is defined as thorns and plateaus which has no relevance at all for voice analysis and is moreover dependent on how the signal is sampled.

25. Explanation: This figure (from the US Patent) is a simple digitization of the incoming signal. Data points whose value is different from flanking data points (presumably at some threshold level) are “thorns”; dats points that are not that different from flanking data points are called “plateaus”. The program computes statistics on the relative incidence of “thorns” and “plateaus” and comes up with a number that is said to show the speaker’s emotional state. The problem (as noted by Eriksson & Lacerda) is that these statistics will vary with sampling rate, quantization of the sampling, background noise, etc. Indeed, it could come up with a “number” just by sampling traffic noise or bird song.

26. From the US patent: This last statement, of course, is utter, unadulterated, nonsense. A student incorporating this in an undergraduate phonetics/speech science course would get an “F”!

27. From the US patent: And yet the Nemesysco website declares their device is not “lie detection” system.

28. “An examination of the description of the method in the American patent documents confirms the suspicion that the method is pure nonsense, perhaps best described as statistics based on digitization artefacts.” [Eriksson & Lacerda]

29. The legal controversy: The use of words such as ‘charlatanry’, ‘fraud’ and, perhaps, the inclusion of reports from a Swedish reporter who interviewed Amir Liberman, the CEO of Nemesysco, revealing that he had no scientific credentials in speech analysis or in psychology or psychiatry, -- in fact no higher education except in marketing -- were grounds for Nemesysco threatening to sue Equinox (the IJSLL publisher) and the authors for libel, defamation of character, etc.

30. Charlantry in forensic speech science: A problem to be taken seriously Anders Eriksson, Francisco Lacerda NOTE FROM PUBLISHER December 4 2008 In the December 2007 Edition of the International Journal of Speech, Language and the Law, an article was published which made serious allegations concerning Mr Amir Liberman and Nemesysco Limited. We have received complaints from Mr Liberman and Nemesysco Limited about the content of this article and particularly that the allegations made against them in it were highly defamatory, containing many inaccuracies and misleading statements. In addition, they complain that it was prepared without reference to them and without giving them an opportunity to comment upon it. The Journal accepts that Mr Liberman and Nemesysco Limited were not asked to assist in the preparation of the article and further that they were not invited to comment on the content of the article prior to its publication where, in view of the content of the article, it would have been appropriate to invite them to do so.We have agreed to publish a letter from Mr Liberman and Nemesysco Limited setting out their objections to the article in more detail in a future issue of the journal. The article will no longer be made available in electronic form through the Equinox website.Janet JoyceManaging Director

31. 3. The response to this controversy in the press and blogosphere.

32. BBC 14 August, 2003 [No author by line] Truth test for insurance claims Banking giant HBOS plans to use voice-sensitive lie detectors in an effort to cut down on fraudulent insurance claims. The technology, developed by Israeli firm Nemesysco Technology and distributed in the UK by security software specialists DigiLog UK, will be used on HBOS' claims hotline for a three-month trial period starting in September. … Managing director Kerry Furber said: … "We're … able to be fairly certain, not 100%, but fairly certain, that there are risk problems within a claim that need further validation. "The psychology of that persuades many claimants to withdraw from the process altogether."

33. Wed December 17, 2008 Behavioral screening -- the future of airport security? By Dana Rosenblatt CNNTEL AVIV, Israel (CNN) -- Keep your shoes and belts on: Waiting in long airport security lines to pass through metal detectors may soon be a thing of the past. Security experts say focus is shifting from analyzing the content of carry-ons to analyzing the content of passengers' intentions and emotions. Nemesysco, another Israeli-based technology company, believes the key to a person's emotions and intentions lies in their voice. The company's patented LVA, or Layered Voice Analysis, technology can pick up verbal cues from a passenger who may pose a threat. Unlike a polygraph test, which checks for lies, Nemesysco's systems work as an "emotion detector," says Nemesysco CEO Amir Liberman. In other words, it's not what passengers say, but how they say it. Nemesysco's devices use a series of patented signal-processing algorithms that can differentiate between a "normal" voice and a"'stressed" voice. If emotional stress is detected, officials can determine if the passenger should be taken aside for further questioning.

34. CNN (continued) The system works on the premise that all voices have a certain frequency, and any deviation of that baseline frequency can indicate stress. Liberman says it takes approximately five to 10 seconds for their system to capture a "normal" voice in casual conversation, which establishes a baseline. Their system then measures changes from the baseline voice that signify an increase in stress, excitement, anticipation, hesitation or other emotions that can indicate a potential terrorism threat. A computer processes the voice patterns and then flashes words such as "high risk," "medium risk," "excited" and "highly stressed." Through his system, Liberman says, he "can see what's going on in your brain." Versions of Nemesysco's system already have been successfully tested at Moscow Domodedovo International Airport, where officials used it to target criminals and drug traffickers. A version was recently implemented at another major international airport which Liberman declined to identify. Layered Voice Analysis also has been used to test for insurance fraud and on the TV program "Big Brother Australia."

35. From a New Yorker piece: HELLO, HAL. Will we ever get a computer we can really talk to? by John Seabrook. JUNE 23, 2008: There is a small market for voice-based lie detectors, which are becoming a popular tool in police stations around the country. Many are made by Nemesysco, an Israeli company, using a technique called “layered voice analysis” to analyze some hundred and thirty parameters in the voice to establish the speaker’s pyschological state. The academic world is skeptical of voice-based lie detection, because Nemesysco will not release the algorithms on which its program is based; after all, they are proprietary. Layered voice analysis has failed in two independent tests. Nemesysco’s American distributor says that’s because the tests were poorly designed. (The company played Roger Clemens’s recent congressional testimony for me through its software, so that I could see for myself the Rocket’s stress levels leaping.) Nevertheless, according to the distributor more than a thousand copies of the software have been sold—at fourteen thousand five hundred dollars each—to law-enforcement agencies and, more recently, to insurance companies, which are using them in fraud detection.

36. http://www.guardian.co.uk/news/2007/sep/09/letters.theobserverhttp://www.guardian.co.uk/news/2007/sep/09/letters.theobserver Letter sent to the on-line version of The Guardian re their uncritical coverage of Nemesysco’s LVA voice stress detector: The truth is out there The technology referred to 'Lie detectors target benefit claim cheats' (News last week) has been subject to considerable testing in the academic research community. It has been found that the results produced by such systems do not exceed the level of chance. But these devices will probably save Harrow Council money because if people are told their speech is being monitored by a gadget that detects lies, they are more likely to tell the truth.Anders Eriksson, Professor of Phonetics, University of GothenburgPaul Foulkes, Reader in Linguistics, University of YorkProfessor Peter French, forensic speech scientist, JP French Associates, YorkFrancis Nolan, Professor of Phonetics, University of Cambridge

37. It is important to note that neither the publisher nor Nemesysco disputed the scientific evaluation of the product. The “sticking points” in the published paper were the words ‘charlatanry’ ‘fraud’ and inclusion of the statements from a Swedish journalist who interviewed Amir Liberman and reported that he had no academic or other scientific credentials which would lend credence to the claims made for the Nemesysco products. Given the plaintiff-friendly libel laws in the UK (it has been characterized as a country that invites “libel tourism”), the publisher did what Francis Nolan characterized as what a person facing a mugger with a gun would do: comply with the mugger’s demands or face devastating consequences. One of the possibly positive outcomes of the Nemesysco scandal is that some MPs and the UK media are urging a change in these libel laws and have cited the Nemesysco case as an example of why this is necessary.

38. 4. Coverage in the ‘Blogosphere’.

39. The response in the “blogosphere” has been harsh – toward Nemesysco – and toward the IJSLL publisher, Equinox (but I don’t think they fully understand the jeopardy faced by publishers given the lop-sided libel laws in the UK). I append to this presentation a number of links to blogs and other websites covering and commenting on this scandal. A purely personal judgment by me – take it or leave it --: the blogs did a far better job of investigating and publicizing this scandal than the established press or TV networks did: CNN, BBC, New Yorker, etc. I tried to get the New York Times interested in this. Result, so far: NOTHING. I also emailed our (Calif’s) senator, Diane Feinstein. Result: NOTHING. If anyone wants to research this scandal: go to the blogs.

40. 5. Who is the guy behind all this?

41. “This controversy, partly fought in a newspaper, caught the interest of a journalist, Arne Lapidius, who was working in Israel for the Swedish daily Expressen. After some research he managed to locate Mr Liberman, a 32 year old (in 2004) businessman in a small office in the town of Natania. The business appeared to be a one-man operation. Mr Lapidus interviewed Mr Liberman about his academic background and was told that he basically had none. He has no degree (never had time to get one, he explains) but has taken some courses in marketing at an Israeli open university. As we have explained above, the LVA is a simple program written in rather amateurishly used Visual Basic.” [Eriksson & Lacerda]

42. Amir Liberman

43. 6. What can be done?

44. Observation: this scandal demeans the speech tech industry. Question: How can the speech tech industry police itself? Suggested answer: the major speech tech organization (ISCA, ASA, ASHA, etc.) have to issue statements emphasizing that claims made about what speech tech systems can do have to be backed up by controlled scientific studies published in reputable peer-reviewed journals.

45. Observation: in matters of food and drug safety and efficacy, in the US, there is the FDA that is a watchdog and gatekeeper on products and processes. Can the speech tech industry lobby for a similar such body to pass on the efficacy of products in their domain. For that matter, why should such a body pass on ANY technical product? Consider, for example, that the NY Times recently reported on a device marketed by a UK company, the ADE 651 “a hand-held "remote portable substance detector" that is claimed to be able to detect from a distance the presence and location of various explosives, drugs, and other substances. The device has been sold to a number of countries in the Middle and Far East, including Iraq, for as much as $60,000 per unit. The Iraqi government is said to have spent £52m ($85m) on the devices. However, investigations by the BBC and other organisations have reported that the device is little more than a "glorified dowsing rod" with no ability to perform its claimed functions. [from Wikipedia]

46. Useful Links.

47. The Nemesysco saga in the News and Blogosphere: http://balneus.wordpress.com/2009/02/17/science-journals-commercial-censorship-law-and-social-security-benefits/ https://antipolygraph.org/cgi-bin/forums/YaBB.pl?num=1230840076/4 https://antipolygraph.org/blog/?paged=2 https://antipolygraph.org/blog/?p=245 Stockholm University account of the event: http://www.su.se/english/about/news-and-events/scientists-threatened-with-legal-action-1.1149 A statement released by Nemesysco – with critical comments by Lacerda attached: http://www.ling.su.se/staff/frasse/LVA_technology2009/NemesyscoResponsetoLacerda_commented.pdf

48. http://languagelog.ldc.upenn.edu/nll/?p=1390 The actual US patent: http://www.freepatentsonline.com/6638217.pdf A brief report by the “Science Insider” (AAAS website): http://blogs.sciencemag.org/scienceinsider/2009/05/royal-swedish-a.html#more “Neuroskeptic” (scroll down to last article): http://neuroskeptic.blogspot.com/2009_02_01_archive.html An account in “Nature News” (web news from the prestigious British science journal, Nature): http://www.nature.com/news/2009/090216/full/news.2009.99.html The Ministry of Truth blog – provides important evidence that the “scientific studies” validating LVA have been done by individuals with a conflict of interest, including financial ties, to firms promoting or distributing Nemesysco products: http://www.ministryoftruth.me.uk/2009/03/18/purnells-lie-detector-indecent-disclosures/

49. http://www.ministryoftruth.me.uk/2009/03/13/purnells-lie-detector-some-things-the-guardian-didnt-mention/http://www.ministryoftruth.me.uk/2009/03/13/purnells-lie-detector-some-things-the-guardian-didnt-mention/ Much overlap with previous links but adds discussion about legal and ethical aspects of the use of Nemesysco’s device and of their threats to quash scientific research which doesn’t boost their sales: http://balneus.wordpress.com/2009/02/17/science-journals-commercial-censorship-law-and-social-security-benefits/ LanguageLog (by Mark Liberman) http://languagelog.ldc.upenn.edu/nll/?p=1390 Language Log (by Mark Liberman) on the history of VSA: http://itre.cis.upenn.edu/~myl/languagelog/archives/001149.html I like this one, “Sceptiphrenia: an occasional blog on science, scepticism, and philosophy” http://sceptiphrenia.wordpress.com/category/voice-risk-analysis/

50. Thank you.