Minerva : A Mixed Reality System for Real-time Construction Problem Resolution

1. Minerva: A Mixed Reality System for Real-time Construction Problem Resolution NSRP BT and SDMT Joint Panel Meeting July 16-18 Cocoa Beach/Cape Canaveral, FL Denny Moore (EB) NSRP Major Initiative Team Lead at Large Phone: 860-867-3700 Email: smoore@gdeb.com

2. Participants and Roles • Technical Contact: Dr. Tom Rando (EB) • NSRP Project Manager: Frances Pearce (ATI)NSRP PTR: Mark Debbink (Newport News)

3. Goal: Mixed Reality for Change Requests

4. Interaction Work Instruction

5. Minerva Demonstration Areas EB BIW

6. Stakeholders Resolving a deckplate issue involves input from a range of stakeholders. Providing an AR tablet to one person could get the trades back to work faster.

7. Workflow The majority of manufacturing questions that arise are resolved through conversations on-site. If a change is required, then planning instigates a CR and derives a CN to update work orders. Engineering support at the site can leverage AR to collaborate with engineers back at the office to fully understand the issue and create targeted solutions faster. {Multiple engagements have been conducted to gather and validate this workflow.}

8. Workflow Scenario A: Deckplate Engineering Support (ES) uses the Minerva AR tablet to video conference with New London Engineering Analysts (EA) to compare AsBuilt conditions with [holographic] 3D CAD in real-time.

9. Workflow Scenario C: After deriving a Change Notice (CN) with a broad range of stakeholders, the intended change is reviewed with the deckplate team – comparing the issue as observed to the proposed change.

10. Interaction • Collaboration between on- and off-site engineers requires a system of devices and applications that: • Provides all necessary capabilities • Is easy to use • The Minerva team of User Experience (UX) designers and hardware/software developers balance: • Which end-user interactions are most valuable • What it is technically feasible/secure {Requirements pertaining to safety, usability, human factors have been captured in correlation with this workflow.}

11. Device Investigation • Microsoft HoloLens • SfB and Skype is discontinued for the HoloLens as everything shifts to the cloud • Use Windows Device Portal to upload our own apps and bypass the store • Toshiba dynaEdge • Skype installed and successfullycalled a desktop user from the dynaEdgeand call recorded • Able to share screens • Latitude 7200 2-in-1 • Same functionality as dynaEdge with a different user experience/interface • Will be tested after purchasing

12. System Architecture

13. Minerva Objectives • The scope of this project entails the use of AR, VR, and MR technologies to substantially improve the process of resolving construction problems in real time. • AR tools support the mechanic in the shipyard • Overlay design objects into partially-constructed environment • Display other design artifacts (work instructions, system diagrams, etc.) associated with components in the work area • Enable the user to identify objects (virtual and physical) for discussion with remote collaborator • Overlay gestures from the remote collaborator • Capture as-built model

14. Minerva AR Activities • Began development of AR registration on HoloLens headset • Adapted UX concepts of registration from ARgos from tablet to headset • Successfully established proof-of-concept export/import of AR map • On hold pending resolution of Skype issue Tablet HoloLens

15. Minerva AR Activities • Worked with UX team on AR interaction modes • Prototype dynamic addition and subtraction of virtual pointers • Will enable the remote collaborator to point to objects in the real world • Evaluating point cloud scanning approaches • HoloLens • DotProduct Dot3D Pro

16. Minerva AR Next Steps • Prototype more types of interaction and data sharing • In-world labels • Geometry • Others? • Investigate ways to interact with documents within tablet AR • Begin work on semantic textures

17. Minerva VR Status – Building Blocks Technical Specifications Quarter 1 Quarter 2 AR Hand Representation Detailed AR Guidance Quarter 3 Quarter 4 Phase 1 Demonstration

18. Minerva VR Status – Progress Technical Specifications Technical Documents Archive with example files and format specifications available on the ATI hosted Minerva SharePoint site.

19. Minerva VR Status – Progress Technical Specifications PLMXML Format

20. Minerva VR Status – Progress Technical Specifications Animation Format

21. Precise Registration - Marker Identification • Initial step for registration • Goal: Identify a fiducial marker (disc)

22. Edge & Contour Detection • Convert to grayscale to simplify image processing operations • several strategies for finding edges: • Initially used only thresholding • Poor robustness to changing light conditions • Contours follow areas of high exposure • Running thresholding after edge detection yields far better results • Settled on canny edge detection then thresholding

23. Ellipse Fitting • The focus of this step is removing false positives quickly: • Initial check: Immediately throw out any contour chain with a low number of points • A poorly detected circle • A false positive • Find ellipses and evaluate bounding boxes for the ellipses • Final check: Remove any bounding box with a high “height-to-width” ratio • An extremely skewed circle • A false positive

24. Detection Robustness • Next tasks • Estimate disc center • Plane segmentation & plane fit • Disc center fit • Gravity vector & uncertainty from IMU • Pose evaluation

25. Questions? Email: DMoore