Development of Machine Vision Technology for Inspection of Railroad Track

1. AAR Technology Scanning Meeting - 24 September 2009 Development of Machine Vision Technology for Inspection of Railroad Track John M. Hart*, J. Riley Edwards, Steven V. Sawadisavi, Esther Resendiz* Christopher P. L. Barkan, Narendra Ahuja* Railroad Engineering Program and *Beckman Institute University of Illinois at Urbana-Champaign

2. Presentation Outline • Background • Prioritization of Inspection Tasks • Current Camera Views • Initial Work on Turnout Inspection • Data Collection • Algorithm Development • Lateral View Processing • Over-the-Rail View Processing • Future Work and Inspection Tasks • Summary and Conclusions

3. Background • Inspection of track and track components is a critical, but labor intensive task resulting in large annual operating expenditures • There are limitations in speed, quality, objectivity, and scope of the current inspection methods • Machine vision provides a robust solution to facilitate more efficient and effective inspection of many track components

4. Machine Vision System Outline Pertinent Information for Railroads Image Acquisition System MachineVision Algorithms Data Analysis System Railroad Track

5. Top Track Related Accident Causes from 2001-2005 (Class I Mains & Sidings)

6. Current Component Inspection Priorities • Rail anchors • Missing • Shifting • Inappropriate patterns • Cut spikes • Missing • Raised • Inappropriate patterns • Turnouts • Missing bolts • Missing cotter pins • Damaged switch and frog points

7. Future Component Inspection Priority • Future inspection tasks: • Turnouts (continuation of current work) • Crossing diamonds • Crib ballast / curve breathing • Other inspection possibilities (may require additional camera views): • Measuring tie spacing • Detecting insulated joint slippage • Monitoring wayside rail lubricator performance • Recording rail manufacturing markings • Determining thermite weld integrity • Monitoring track circuit bond wire condition • Inspecting grade crossing surface condition • Measuring flangeway width and depth

8. Current Camera Views: Over-the-rail View • Used for measuring the distance between the spike head and the base-of-rail and verifying spiking patterns • Also used to measure crib ballast level with respect to the top of the tie

9. Current Camera Views: Lateral View • Optimal view for measuring the distance between the rail anchors and the edge of the ties and verifying anchor patterns • Used to verify spike heights measured in the over-the-rail view • Conducive to panorama generation

10. Turnout Background - FRA Data Analysis • According to the FRA (Federal Railroad Administration) Accident database, the top three causes of turnout-caused derailments are: • Broken or Worn Switch points • Broken Switch • Switch point gap • Switch points account for almost 13% of all rail accidents • Frogs account for about 3%, in particular, worn or broken frog points

11. Turnouts: Areas of Focus • Switch • Distance of switch points from stock rail when open and closed • Missing bolts and cotter pins • Worn points • Frog • Worn or broken point • Guardrail • Gauge check clearance

12. Field Data Collection • Initially used handheld cameras to take images • Obtained insights on later challenges, such as variability in component appearance • Developed Video Track Cart for field data acquisition • Used on low density track for verifying camera views and experimenting with lighting • Optimizing lighting is a challenging task • Reviewed lighting solutions from existing machine-vision systems

13. Video Track Cart • Equipped with two camera mounts for the over-the-rail view and a gauge-side lateral view • Geared tripod heads for precise adjustment of camera views • Deep-cycle marine battery supplies power for all electronics • Rugged laptop resistant to environmental conditions

14. Initial Field Video Data Collection • Visit to Advanced Transportation Research and Engineering Laboratory (ATREL) in Rantoul, IL - Tested on track panel • Monticello Railway Museum (MRM) visits • Improved lateral view and over-the-rail view mounts • Manually measured and catalogued track defects for algorithm verification

15. NS Field Video Data Collection Collected 20 videos on NS track west of Champaign, IL using both camera views Components captured: Turnout, panelized concrete grade crossing, continuously welded rail (CWR), tangent and curved track, anchors, and cut spikes Captured videos under a variety of lighting conditions

16. Lighting Considerations • Reviewed lighting systems of similar machine-vision systems • University of Central Florida’s track inspection system • Lasers and timed strobe lights • Sun shields • ENSCO’s joint bar inspection system • High-powered xenon lights • No sun shields • Georgetown Rail’s Aurora system • Lasers with light filters on cameras • Lighting solutions are unique to each system’s data collection requirements, with our system requiring even illumination over a large area of track

17. Track Component Isolation Delineate ties, ballast, and rail using edge detection Incorporate texture classification to make edge detection robust Extract patches of texture from a hand-labeled training image Compare new texture patches against known textures Use Gabor filter outputs to classify patches from new video frames Edge image (base-of-rail highlighted) Original image Tie patches Ballast patches Rail patches

18. Base-of-Rail Delineation • The original image, where we will isolate the base of the rail

19. Base-of-Rail Delineation • Strong horizontal edges are detected (candidate edge is shown)

20. Base-of-Rail Delineation • Surrounding pixels above and below candidate edge are examined

21. Base-of-Rail Delineation Ballast below edge Ballast below edge • Base of the rail is confirmed

22. Component Detection • Detect tie plates using similar edge/texture method • Edge detection, verified with texture classification • Hypothesize the spike locations (and missing spike locations) using prior knowledge of tie plate structure • Spikes (and missing spikes) are located with template filters • Spike height is measured relative to the base of the rail Delineated components Original Image Spikes and missing spikes

23. Component Detection: Anchors • Detect anchors • Use prior knowledge of anchor location (since tie plate and rail bottom have been delineated) • Find parallel edges to achieve anchor detection • Robust to changes in anchor orientation Isolated anchors

24. Defect Detection Results Individual components are located and measured Tie with shifted anchor

25. Defect Detection Results Individual components are located and measured Tie with shifted anchor

26. Defect Detection Results - Shifted Anchor Defect detected if anchor displacement with respect to the edge of the tie exceeds a threshold Tie with shifted anchor

27. Component Detection on Field Video Field data video

28. Panoramic Generation

29. Track Panorama Visualization of cumulative track components using panoramic stitching Tie / Tie plate delineation on test panorama Component detection on test panorama

30. Performance • Lateral-view video from NS track (54 consecutive ties) Tie and anchor detection rates (precision and recall)

31. Performance • Lateral-view video from NS track (54 consecutive ties) Tie plate hole and spike localization success

32. Over-the-rail Inspection - Original Frame • Locate track components from over-the-rail video • Original frame

33. Over-the-rail Inspection - Rail Location • Locate rail by spatial self-similarity across consecutive frames • Locate rail

34. Over-the-rail Inspection - Texture Classification • Classify texture in frame as ballast/non-ballast • Detect ties with tie filter • Tie Filter • Tie Filter • Locate ties with filter

35. Over-the-rail Inspection - Tie and Plate Location • Localize tie and tie plate using edge boundaries • Ballast detection used as preliminary step • Delineate ties with gradients

36. Over-the-rail Inspection - Spike Location • Locate spike and spike hole with spatial template • Locate spike and tie plate hole

37. Over-the-rail Inspection Results • Inspected frame

38. Over-the-rail Inspection Video • Field data video

39. System Implementation • Future system likely to be mounted on either: • Track geometry car • Rail defect detector car • High-rail vehicle • Possible initial implementation on defect detector car • Inspections occur with acceptable frequency • Crews better trained with advanced inspection equipment • Already equipped for massive data storage and upload • The system will be initial adapted for experimental testing high-rail vehicle

40. Future Work • Plan for the remainder of 2009 • Continue testing spike and anchor algorithms on field-acquired video • Begin lighting experimentation on Video Track Cart • Complete study on methods to inspect for turnout defects • Develop an approach for detecting turnout components • Record additional videos at MRM and on other local tracks • Approach for 2010 • Investigate cameras which can run at higher speeds and in a greater variety of environmental conditions • Develop and test algorithms for turnout defects • Begin adaptation of acquisition system for trials on a track vehicle

41. Future Component Inspection Discussion • Future inspection tasks: • Turnouts (continuation of current work) • Crossing diamonds • Crib ballast / curve breathing • Other inspection possibilities (may require additional camera views): • Measuring tie spacing • Detecting insulated joint slippage • Monitoring wayside rail lubricator performance • Recording rail manufacturing markings • Determining thermite weld integrity • Monitoring track circuit bond wire condition • Inspecting grade crossing surface condition • Measuring flangeway width and depth

42. Acknowledgements 42 • Research Sponsors • AAR Technology Scanning Program • NEXTRANS Center • Industry Assistance • David Davis of TTCI • Technology Scanning Committee • CN • UP • NS • BNSF • Monticello Railway Museum

43. UIUC Multidisciplinary Research Team Computer Vision and Robotics Laboratory: Narendra Ahuja, Professor, Beckman Institute John M. Hart, Senior Research Engineer, Beckman Institute Esther Resendiz, Graduate Student, Electrical and Computer Engineering Railroad Engineering Program: Chris Barkan, Associate Professor, Director of Railroad Engineering Program Riley Edwards, Lecturer, Railroad Engineering Program Luis Fernando Molina Camargo, Graduate Research Assistant Matt Toussaint, Undergraduate Research Assistant Randy Nolden, Undergraduate Research Assistant Brennan Caughron, Undergraduate Research Assistant

44. Appendix Additional Turnout Data

45. Turnouts: Switch Point Inspection • Rail point closed tightly • Chipped • Gap on open rail point is wide enough for wheel flange • Missing cotter pins

46. Spring Frog Inspection • Missing cotter pins • Gauge • Springs working • Tightly fit

47. Bound Manganese Frog (BMF) Inspection • Gauge • 6” of point missing • worn down 5/8” • Flangeway depth • Worn tread point 6” max 4” defect