Validating the use of Handwriting as a Biometric and it’s Forensic Analysis

1. Validating the use of Handwriting as a Biometric and it’s Forensic Analysis Graham Leedham & Vladimir Pervouchine C2i, School of Computer Engineering Nanyang Technological University Singapore

2. Structure of this Talk • Brief History of Handwriting • A look at the Variability of Handwriting • A look at Forensic Document Analysis • Computer Tools to assist FDE’s • Is handwriting an accurate biometric? • Study of the effectiveness of features used by FDE’s

3. History of Handwriting • 25000 years ago • Cave paintings are the oldest pictures ever found. Many were made more than 25,000 years ago by 'stone age' cave dwellers using sticks, sharp stones or their fingers. For 'paint' they used charcoal, coloured earth and vegetable dyes. • Early man could not write, so to remember things or leave messages he drew pictures on cave walls, rocks, bones or wet clay. Gradually, over the years, pictures became symbols, and then letters to form alphabets of signs to represent sounds.

4. History of Handwriting • SUMERIAN CUNEIFORM • Some of the earliest examples of a writing system come from the Sumerian people who lived in the Middle East between 4000 and 6000 years ago. • 'Cuneiform' means 'wedge-shaped' because the inscriptions were made by pressing the triangular tip of a reed or a stick (stylus) into wet clay tablets. The wedge marks were combined into signs representing objects and ideas. At first there were over 2000 different signs, but the Sumerians gradually reduced their 'alphabet' to about 600 symbols. It was also at about this time that Chinese characters began to emerge independently in China with symbols being written on bone and shells. They too were originally pictures and symbols to represent ideas and objects. These were the earliest forms of writing. And subsequently had its own history of development.

5. History of Handwriting • 4000 years ago • EGYPTIAN HIEROGLYPHICS • While cuneiform was spreading throughout Mesopotamia, a different writing system was being developed in nearby Egypt. From about 5000 years ago the Egyptians used a form of stylised picture writing called hieroglyphics. ('Writing of the Gods'.)

6. History of Handwriting • 2500 years ago • GREEK • About 2500 years ago, the ancient Greeks were using an alphabet very much like our own. In fact, the word 'alphabet' comes from the first and second Greek letters, 'alpha' and 'beta'. • The Greek alphabet was developed from the Phoenician writing system. The Phoenicians were great sailors and merchants who traded with many countries in the Mediterranean, taking their writing with them. But the Greeks added signs for vowels because the Phoenician alphabet contained only consonants.

7. History of Handwriting • 2000 years ago • ROMAN • When the Romans conquered Greece just over 2000 years ago they took over the Greek alphabet and altered the shape of many letters. The letters of the English alphabet come directly from the Roman alphabet, although we have added three extra letters: J, U and W.

8. History of Handwriting • 800 years ago • GOTHIC • After the Roman Empire collapsed in the 5th century AD, it was mainly monks who kept up the art of writing. Soon every monastery had its own scriptorium where manuscripts were copied, decorated and bound into books.

9. The Magna Carta written in 1215(written in Latin)

10. History of Handwriting • 500 years ago • ITALIC • In the 15th century a group of Italian scholars working in Florence decided that Gothic was difficult to read so they developed a new script. This style soon became popular all over Europe. Even today we still call styles like this 'Italic' because they came from Italy.

11. History of Handwriting • 300 years ago • COPPERPLATE • The 'Copperplate' style of handwriting was taught by writing masters from the 17th century onwards. Sometimes known as the English running hand, this neat easy-to-read script was also easy to write. Word after word could be produced without lifting the pen between letters. Until the invention of the typewriter it was widely used for business records and legal documents.

12. History of Handwriting • Several handwriting styles are now taught in schools around the world. There is less variation in style.

13. Variability of Handwriting Individual styles develop

14. Variation of the word “the” written by 8 different writers. Source: Harrison, 1981

15. Variation of the letters “G” and “R” written by 15 different writers. Source: Harrison, 1981

16. Example of variation in letter formation styles in 10 letters from 9 different writers. Source: Harrison, 1981

17. Handwriting can be produced by many different writing instruments

18. Handwriting has been used as a legal or official seal for centuries “Set your hand to the document.” “Make your mark.”

19. Forgery / Disguise / Alteration • Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or • Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or • Is the writing ALTERED? (Has someone modified or altered the original document?)

20. Recognition for machine transcription Writing analysis for Authentication Mathematical formulae Printed characters Numerals Alphabetic characters Symbols Signature verification Writer identification Forgery identification Disguised writing Cursive script Whole words Separate characters Hierarchy of Handwriting Recognition Problems (OFF-LINE & ON-LINE) Automatic processing of handwritten documents

21. Current Methods used by Forensic Document Examiners • Primarily involves manual extraction and comparison of various global and local visible features. • They are usually doing a comparison test between a “Questioned Document” and a set of “Known Documents”. • The objective is to determine whether the “Questioned Document” was, or was not, written by a particular individual. • The “Questioned Document” may be in disguised handwriting.

22. Current Methods used by Forensic Document Examiners • Global features: Handwriting size, word spacing, line spacing, • arrangement of words, margin patterns, baseline patterns, • line quality, spelling, grammar … • Local features: character size, height-width ratios of characters, • Size and shape of loops, letter slant, letter design, letter spacing, writing pressure, • speed variation, t-crossings and i-dots, hooking, punctuation marks ... FOR MORE INFO... See texts: Harrison W.R. (1981), Suspect Documents, their Scientific Examinations, Nelson-Hall Inc., Illinois. Hilton O. (1993), Scientific Examination of Questioned Documents, CRC Press, Florida.

23. Current Methods used by Forensic Document Examiners Hidden writing left as pressure indentations on sheets below the one written on are recovered using ElectroStatic Detection Apparatus (ESDA) equipment.

24. Current Methods used by Forensic Document Examiners • FISH - Forensic Information System for Handwriting - used by the bundeskriminalamant, Germany to maintain a database of known and unknown writers. A handwriting sample is characterised by interactive extraction of several global and local features to create a database of handwriting which can be indexed to locate similar handwriting to that in a Questioned Document.

25. Current Methods used by Forensic Document Examiners • Tick sheets, or comparison charts, (used in the UK and other countries) are created showing side-by-side comparison of known and questioned words or characters. The comparison is subjective and based on local and global features. The degree of similarity is graded on a five point scale – 1. Was written by…., 2. High probability it was written by...., 3. Probable/could well have been written by…., 4. No evidence, 5. Inconclusive.

26. Example of a manually produced comparison chart to show that these writings were produced by different writers: Source: Harrison, 1981

27. Forgery / Disguise / Alteration • Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or • Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or • Is the writing ALTERED? (Has someone modified or altered the original document?)

28. Example of a manually produced comparison chart to show disguised handwriting. Source: Harrison, 1981

29. Forgery / Disguise / Alteration • Is the writing FORGED? (the author is not who he claims to be and is attempting to assert the writing is the same as someone else’s) or • Is the writing DISGUISED? (the author wishes to deny doing the writing at a later date) or • Is the writing ALTERED? (Has someone modified or altered the original document?)

30. EARLY COMPUTER TOOLS TO ASSIST DOCUMENT EXAMINERS: The early computer tools were predominantly in the use of the COMPUTER IMAGE PROCESSING and image handling techniques. e.g. Behenen and Nelson, 1992. And in the ENHANCEMENT of poor images as encountered in ESDA lifts or provide alternative methods to view the hidden writing. e.g. Baier et al., 1987 As well as a number of attempts at WRITER IDENTIFICATION e.g. Kuckuck et al., 1979

31. Our Earlier Research attempted to ease the workload of the FDE Objective: to investigate the possible use of computer based image handling tools to assist a document examiner in the analysis of a handwritten document and preparation of the evidence. 1. Oct 1993 - Dec 1994: (FODES) Holcombe G., An experimental image processing environment for the forensic examination of questioned documents, MSc Dissertation, University of Essex, 1995. 2. Oct 1994 - Dec 1997: (WIS) Greening C., Automatic writer identification for forensic document analysis, PhD Dissertation, University of Essex, 1998. 3. May 1996 - April 1999: (FOX) Applied Research Fund Project (RG25/95) “Image Analysis Tools for Authentication and Enhanced Classification of Handwritten Script Using Forensic Techniques” carried out at Nanyang Technological University.

32. Previous Research at NTU (& Essex University) Achievements: Prototype system comprising tools to: 1. Scan and view images, segment text from background. 2. Various automatic and interactive tools to process the handwritten documents: line, word and character segmentation, chart generation, slant variation, loop, ascender and descender feature extraction, animated word, character or signature visualisation, stroke sequence extraction ….

33. scanned image • 3. Feature Extraction: • Baseline patterns and angle • Word slant • Average stroke width • Height of main body, ascenders • Depth of descenders • Loop features: area, slant, • circle dissimilarity index various features • 1. Handwriting Extraction : • Background removal • Shadow noise removal • Salt and pepper noise removal noise free image simulated images • 4. Simulation : • Ruler guided writing • Slant manipulation • 2. Line, Word, Character • Segmentation: • Text is available TOOLS TO ASSIST DOCUMENT EXAMINERS: Work at Essex University and Nanyang Technological University: Leedham, Sagar, Solihin, Holcombe, Chong, Greening (1993-2000) FOX & WIS systems Purpose: To enable document examiners to produce display charts and physical measurements.

34. Previous Research Results Achieved Screenshot of visualizationsoftware • Supplementary software for visual comparison of letter formation has been implemented. • A set of rule-based algorithms have been developed for feature extraction.

35. TOOLS TO ASSIST DOCUMENT EXAMINERS: Work in Germany & Netherlands: BKA, Schomaker, Franke, de Jong, et al. 1994 - One of the first systems to be installed to provide database matching of handwriting features for individuals was the FISH system introduced by the German law enforcement agency (Bundeskriminalamt) which semi-automatically extracts features to characterise handwriting and store them in a database. (Shown at the 5th IGS, 1991, Tempe, by Manfred Hecker) A recent research consortium has sought to extend this work in the WANDA system (see http://www.ai.rug.nl/alice/wanda/)

36. TOOLS TO ASSIST DOCUMENT EXAMINERS: Work in Australia: Found, Rogers, Sita et al. 1995 - Investigating handwritten signature complexity and tools to assist document examiners. Providing objective means for the subjective techniques employed by document examiners. Other work: There are numerous other researchers currently working on writer identification and tools to identify forged handwriting: Eg. Bensefia et al. from Rouen University, France Fairhurst et al. from Kent University, UK Cha & Tappert from Pace University, USA Ueda et al., Nara College of Technology, Japan

37. What is the problem? So what is all this about??? People have been doing HANDWRITING for 1000’s of years, Crime involving FORGED, DISGUISED or ALTERED handwriting is common in cheques, anonymous letters, wills and other examples where deception can lead to illegal gain or advantage. FORENSIC DOCUMENT EXAMINERS have been authenticating handwriting in one form or another for more than 100 years. Many TEXT BOOKS have been written describing the analysis techniques and providing CASE STUDIES. Today all LAW ENFORCEMENTS AGENCIES practice Questioned Document Examination and employ Forensic or Questioned Document Examiners along with other Forensic Scientists. There exist a number of PROFESSIONAL ORGANISATIONS around the world which foster professional training and accreditation to forensic Document Examination.

38. THE PROBLEM IS….. Other branches of Forensic Science such as DNA analysis, blood, fibre and soil analysis are supported by a wealth of CHEMICAL, BIOLOGICAL and PHYSICALKNOWLEDGE obtained from years of scientific research and published in learned journals. Forensic Document Examiners do not have any similar SCIENTIFICBASIS to support their EXPERTOPINION. The current acceptability of Forensic Document Examiners expert opinion is based on the CREDIBILITY, EXPERIENCE and STANDING of the FDE. They have little scientific evidence to support their opinion. Their judgement is subjective and not an exact science. Recent legal challenges (eg Daubert, 1993 and Starzecypzel, 1995) have brought this lack of scientific support into the limelight.

39. OBJECTIVES…… • During the past 20 years there has been sporadic interest in the application of computer systems to ASSIST AND SUPPORT the work of Forensic Document Examiners. • In this rest of this presentation some of our current research work is presented which: • Provides computer support to the analysis methods used by document examiners • Seeks scientific evidence to support the analysis methods used by forensic document examiners. Whilst much of the research carried out in handwriting processing and recognition provides considerable scientific knowledge about the education of writing styles, cognitive processes and motor-control activities in the production and recognition of handwriting, only a limited amount of the work is directly applied to the work of the FDE.

40. Research Motivation (restating) Most of the techniques employed by document examiners are based on standard practices and previous experience. The Forensic Document Examiner as an Expert Witness is highly reliant on their personal standing and credibility. There is very little scientific justification for many of their practices and procedures. The only truly scientific examination is the chemical analysis and dating of ink and paper. Several court rulings have brought into question the scientific basis of the expert document examiners testimony. Eg United States vs Starzecpyzel, 1995

41. Is handwriting a biometric? • The argument is that - because handwriting is a practiced ballistic skill, it represents some uniqueness to the individual which can be used to identify the individual. • However skilled forgery is widespread, and individual variability due to health, stress writing implement, writing stance etc… is significant. Where does the genuine end and the forgery begin?

42. APPROACH 1. Is handwriting unique? APPROACH 2. Are professional forensic document examiners better than other people at handwriting examination? APPROACH 3. Is it possible to verify the effectiveness of features employed by forensic document examiners? Feature extraction: Proves it is a biometric. Mainly computational features + some document examiner features Justifies the need for tools to assist forensic document examiners in what they do. Feature extraction: Justifies the use of features/methods used by Document Examiners. CURRENT APPROACHES Scientific basis for forensic document examination

43. Handwriting Sample Identification Algorithm Writer 1 Writer n (a) Verification Algorithm Same Writer Handwriting Sample 1 Different Writer Handwriting Sample 2 (b) APPROACH 1: IS HANDWRITING UNIQUE? Work on individuality at CEDAR (Srihari et al. Journal of Forensic Sciences, 2002) • Establish discriminatory power of handwriting • Use objective methods • Algorithms suitable for software implementation • Relate methods to FDE (forensic document examination) procedures • Rigorous testing, establish error-rates • Peer-review Two tests for establishing error rates

44. APPROACH 1: IS HANDWRITING UNIQUE? POVIDING TOOLS TO ASSIST DOCUMENT EXAMINERS: CEDAR-FOX toolset developed at CEDAR

45. APPROACH 2: ARE PROFESSIONAL FORENSIC DOCUMENT EXAMINERS BETTER THAN OTHER PEOPLE IN HANDWRITING EXAMINATION? Work at Drexel University, USA: Kam et al (1994- ) and Work at LaTrobe University, Australia: Found, Rogers et al (1994-) Both groups performed a number of experiments comparing professional FDE’s and lay people. They concluded that “Professional document examiners DO possess writer-identification skills absent in the general population.”

46. APPROACH 3: IS IT POSSIBLE TO VERIFY THE EFFECTIVENESS OF FEATURES USED BY FORENSIC DOCUMENT EXAMINERS? Handwriting has been shown to be discriminative when identifying whether an unknown writer is one a group of N known writers and verifying whether two documents are from the same or different writers. (Srihari et al) Document examiners have been proven to be better than lay people at writer identification/verification, including forgery identification. (Kam et al & Found, Rogers et al) The methods employed by document examiners are well documented in various texts.

47. APPROACH 3: IS IT POSSIBLE TO VERIFY THE EFFECTIVENESS OF FEATURES USED BY FORENSIC DOCUMENT EXAMINERS? • The comparison methods used by FDE’s are frequently QUALITATIVE /SUBJECTIVE and the resulting evidence also qualitative / subjective. • QUANTITATIVE / OBJECTIVE ANALYSIS of the methods used by FDE’s is necessary to determine techniques and methods which can be supported by scientific proof. This is what we are trying to do.

48. APPROACH 3: OBJECTIVES OF THE STUDY 1. To develop a system to automatically extract structural features from individual handwritten characters. 2. To assess the uniqueness and individuality of the structural or visually observable features used by Forensic Document Examiners 3. To determine whether the features are unique to a writer and can be used for author identification

49. SELECTION OF CHARACTERS TO STUDY Cannot examine all characters as feature sets must be individually tailored and algorithms written: • Choose frequently occurring characters • Characters must potentially possess writer-specific features. (Capital letters as well as letters that consist of several strokes, like those with ascenders or descenders, bear more individual information than simple characters like “i” or “c”.) • Robust automatic feature extraction of the writer-specific features must be achievable Based on an analysis of 220,000 words from three novels, the characters chosen for analysis were ‘d’, ‘y’, ‘f’ and grapheme ‘th’. The three letters and one grapheme represent >12% of typical script.

50. Frequencies of letters and graphemes in English texts • Letters / graphemes should: • be frequent • have ascenders / descenders • not have too simple shape