ASL2TXT

1. ASL2TXT Converting sign language gestures from digital images to text George Corser

2. Presentation Overview • Concept • Foundation: Barkoky& Charkari (2011) • Segmentation • Thinning • My Contribution: Corser (2012) • Segmentation (similar to Barkoky) • CED: Canny Edge Dilation (Minus Errors) • Assumption: User trains his own phone

3. Concept • Deaf and hearing people talking on the phone, each using their natural language • Sign-activated commands like voice-activated

4. Situation: Drive Thru Window Think: Stephen Hawking Deaf person signs order Phone speaks order Confirmation on screen

5. Process Flow • Requires several conversion processes • Many have been accomplished • Remaining: ASL2TXT

6. Goal: Find an Algorithm • Find an image processing algorithm that recognizes ASL alphabet = A Web site

7. Barkoky: Segmentation & Thinning Barkoky counts endpoints to determine sign (doesn’t work for ASL)

8. Barkoky Process Segmentation Thinning Input: hand segment Apply thinning Find endpoints, joints Calculate lengths Clean short lengths Identify gesture by counting endpoints • Capture RGB image • Rescale • Extract using colors • Reduce noise • Crop at wrist • Result: hand segment

9. 1. Capture RGB Image2. Rescale % ---------- 1. Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % ---------- 2. Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10. 3. Extract Hand Using Colors % ---------- 3. Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; end; end; figure('Name','Extracted'),imshow(abw10); Note: Color threshold code differs from Barkoky

11. Colors: Training Set Histograms

12. Colors: Training Set (2) Red Green Blue Excel

13. Colors: Test Set Histograms

14. 4. Reduce Noise % ---------- 4. Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, abw10(i,j,1)=0; end; end; end; end; abw10 = imfill(abw10,'holes');

15. 5. Identify Wrist Position % ---------- 5. Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break; end; end;

16. Wrist Detection • Algorithm searches bottom-to-top of image • Finds a leftmost white pixel above black pixel • Sets wrist position SE of found white pixel

17. Corser: Segmentation & CED • Segmentation (similar to Barkoky) • Color threshold technique slightly different • American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers • Image Comparison: Tried Several Methods • Full Threshold (Minus Errors) • Diced Segments (Minus Errors) • Endpoint Count Difference • CED: Canny Edge Dilation

18. ASL Training Set Hit-or-miss: 23% Barkoky: 8%

19. ASL Test Set MATLAB

20. A

21. A

22. B

23. B

24. C

25. C

26. D

27. D

28. E

29. E

30. F

31. F

32. G

33. G

34. H

35. H

36. I

37. I

38. J

39. J

40. K

41. K

42. L

43. L

44. M

45. M

46. N

47. N

48. O

49. O

50. P