The Portable Sign Language Translator (PSLT): from wishful thinking to commercial product

1. Ernesto Compatangelo, Ryan RussellPSLT project (http://www.pslt.org) Technabling Ltd PSLT development is funded by the Small Business Research Initiative (SBRI) sponsored by the Department for Business Innovation and Skills (BIS) and the Technology Strategy Board (TSB) and managed by JISC TechDis, a leading UK advisory service on technology and inclusion. The Portable Sign Language Translator (PSLT):from wishful thinking to commercial product

2. The PSLT in a nuthell • NOW: translates sign languages (BSL, CSL) into text; • 2014: text into sign languages (using an Avatar) • NOW: uses Customisable SL (CSL) to control devices

3. Talk structure • Motivations and background • Relevant features • A short demo of the PSLT in action • Technical challenges • Social, cultural, and mundane issues • Wider applicability of video-based analysis

4. Motivating the PSLT (1) • There are around 100,000 Sign Language (SL) users in the UK only (mostly BSL); millions worldwide • Profoundly deaf SL users only communicate w/ sign • Most Voice & Text (V&T) users do not sign • No signer-to-speaker communication gaps in order to remove educational, workplace, lifestyle barriers

5. Motivating the PSLT (2) • Sign interpreters are few and expensive – only use when strictly needed • A tool to self-learn SL: low cost, anywhere, any time • Enabling independent life means allowing gesture and vocabulary customisation, not just translation

6. Motivating the PSLT (3)The Small Business Research Initiative (SBRI) • UK Gov needs new products for wider society • 1 - Contract awarded to small businesses for R&D • 2 - Follow-up for market-ready commercial product • Cost-effective, delivering value for money product • Developed technologies owned by businesses • NOT for the academic sector

7. Requirements & desiderata • Sign to text (the real challenge) - no text to sign • Works on “mobile” technology • Open source components only – no strings attached • Output: text/sound + execution of commands • Discreet, unobtrusive, flexible, to run on std HW

8. NO-GO areas (1) • What about these as a translation device? Signing gloves for computerised sign-to-text translation

9. NO-GO areas (2) • What about this as a translation device? Prototype device for gesture control

10. NO-GO areas (3) • What about this as (part of) a translation device? Kinect, a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs

11. Core PSLT feature • No weird/costly devices – just off-the-shelf HW • Translates BSL + CSL (combined) + Makaton into text • Has provisions for other SLs & regional variations • Fully customisable: users can create their own signs and give them specialist meanings not in other SLs • Works on many devices, from smartphones to PCs • Signs translated into ambient device commands

12. Time for a demo... the PSLT in action

13. Problems tackled: what others avoided to consider • Dynamic background: light, movement • Ind. form factors: skin, body shape, signing style • The inherent fuzziness of the signing process • Recognise & disambiguate ALL hand gestures (~1200) • Individual signs, profiles, & regional variations • Create your gestures with your specialist meaning

14. The challenges of image recognition • Identify & track moving shapes in the “foreground” against a noisy background with sudden light changes(the signer does also move while signing!)

15. The technical challenges • Recognise (i) individual signs and (ii) hand gestures • Disambiguate meanings if gestures are overloaded • Articulate adverbs, verbs, adjectives, subjects, objectsin a grammatically correct and meaningful sentence • Bridge SL syntax with written text syntax • Run under Win, Linux, Android with basic HW

16. Typical objections (not from real users) • It can't be done - Academic linguists • It won't work – there is “much more” to SLs than just hand gestures (e.g., facial expression) - Academic linguists + worried interpreters • You are offending the sign language – as above • Money spent on technology should be better spent on extra interpreters – interpreters, social workers– some members of “the voice of deaf people” + interpreters Can you spot vested interests / marginalised complainers?

17. The position of “real” stakeholders • “PSLT has the potential to revolutionise the way deaf and hearing impaired individuals can communicate in every day situations that hearing people take for gramnted” - Jon Smith, Vice Principal, Communication Specialist College, Doncaster • “Not having to rely on an interpreter all the time would be good” - Young learner, Communication Specialist College, Doncaster • “I can see the benefits and impact rhis will have on my ability to communicate in a range of community settings” - Brian Kokoruwe, inclusion consultant and BSL user

18. The near future • Facial expressions (technology there, needs time) • Performance (speed), particularly on Android • Automated profile upload/download • BSL word usage frequencies for overloaded gestures(joint project with UCL, end of 2013) • Text-to-sign Avatar (proof of concept successful)

19. Thank you for your time • Any questions? • Please ask

20. Other relevant PSLT features • Gestures can have multiple meanings: signers choose • Text sent to PSLT-hosting display or other devices • Avatar for text-to-sign available on smartphones too • Signs translated into ambient device commands andlocal/remote appliances (via Bluetooth, WiFi, GSM) • Future: voice rendering of signed text & voice input