Customizable Embedded 3D Surround View Turn-Key Solution on Apalis iMX6 SoM

1. Customizable Embedded 3D Surround View Turn-Key Solution on Apalis iMX6 SoM - Author: Thomas Hase, TES Electronic Solutions GmbH

2. Multi-Camera 3D Surround View systems are already established in many passenger vehicles. Demand for such systems now also grows for special vehicles like construction machines, harvesters, buses or military vehicles. In comparison to personal cars, these vehicles are produced in a rather moderate volume and have a high need for customization of the optical system.

3. In this blog post, we present a customizable 3D Surround View Turn-Key solution based on the Toradex Apalis iMX6 SoM based on the NXP® i.MX 6 SoC, that can be quickly adapted concerning the specific requirements of the targeted vehicle.

4. Please watch this video from Embedded World 2017 in Nuremberg, Germany, where this system solution was presented: Video Link: https://www.youtube.com/watch?v=Y67DhP6qaUs

5. We also explain how to get familiar with the topic of 3D surround view, evaluating the solution via a step-by-step approach and finally how to come to a product meeting your specific needs. TES 3D Surround View Stitching Library – the heart of the system solution TES Electronic Solutions provides a 3D Surround View software library designed for embedded systems, CPU/GPU independency, and high customizability. Detailed information can be found on the TES 3D Surround View Website.

6. The library takes raw fish-eye pictures from memory and computes a 3D View of the vehicle in its maneuver situation. • TES 3D Surround View Highlights: • Increases safety and comfort through improved situation awareness: • Excellent stitching and 3D scene generation • Watch your maneuver situation from any position outside or inside the vehicle • Any number of predefined viewing positions (“Views”) possible with animated transitions between different views • Many parameters configurable per “View” • Highly customizable and portable • Independent from the used type of cameras (the software expects raw fish-eye pictures in memory)

7. 3D Surround View Turn-Key Solution In the following, we present a complete HW/SW system solution based on the Toradex Apalis iMX6Q IT Module allowing easy hardware customization to meet specific customer demands. System Setup

8. The System setup consists of the following components: • Processor: Toradex Apalis iMX6 IT Computer On Module with Heatsink • Toradex Ixora Carrier Board • Camera System: • First Sensor Mobility Area View Kit – DC–AVK including • 4x 1MPixel MJPEG Ethernet Cameras (PoE), DC3K-1-EAP • 4x 5m Ethernet cables (up to 25m possible for each camera) • Micrel Gigabit Ethernet (PoE) Switch • Metal Plate for Desktop Application, additionally 4 Vacuum cups for vehicle application included • Software: • TES 3D Surround View Software under Linux on Apalis iMX6

9. Resource Usage: • The 3D Surround View algorithm mainly exploits the GPU. The CPU is only used for system control and in the case of view-changes. • Configuration: 4 MJPEG IP cameras, combined 80Mbit data rate @30 fps and 1Mpx resolution. Low polygon car model, 1280x1024 screen resolution at 60Hz refresh rate. • GPU: 3-6 GFlops (proportional to pixel amount depicted on screen) • RAM usage: 60 MB in total • 4* 10 MB for camera data buffers • 2 MB for 3D environment and car model • 18 MB for camera video textures

10. This turn-key system solution can be rapidly reproduced for evaluation and prototyping purposes. Of course, also the separate components can be exchanged or adapted to the target application needs, e.g.: • The cameras can be exchanged against other types. • A dedicated application specific main board can be designed taking the Ixora Carrier board as a reference. • The user interface and HMI overlay of the 3D Surround View software can be easily adjusted. • The 3D model of the vehicles can be easily exchanged. • Both Toradex and TES, are open to provide the necessary support.

11. Getting started and familiar with 3D Surround View • 3D Surround View is not only a topic of running an algorithm on an embedded platform and be done with it. There are many aspects of the system setup that need to be considered and require customization for specific use cases, e.g.: • Selection of the appropriate cameras regarding lens, resolution, physical data transmission (cable length!), power supply, number of cameras etc. • Positioning of the cameras (usually the higher the better) • Defining the appropriate viewing positions, i.e. virtual position and viewing vector of the “observer” in the 3D scene • Optimizing the 3D Surround View parameters, e.g. Environmental model (shape and size of the projection bowl), stitching method and parameters. • User interface for the 3D Surround View System • 3D Model of the vehicle, potentially with transparency.

12. Just consider an excavator compared to a personal car: On an excavator, you need significantly longer cables to the cameras, the cameras are mounted typically much higher. Latency is not that much of a concern as the driving speed is lower and maybe you just want to apply only 3 cameras or mount the cameras un-symmetrically because the driver can see the left front part of his vehicle very well while he cannot see the right front part at all. So it might be a good idea to first “play” with a 3D Surround View system to get an idea of such application aspects. For this purpose or for those who just want to give the TES software a try there is an offline PC demonstrator available at zero cost. TES provides this demonstrator on request in a zip archive with no need of installation (just unpack). Any modern Windows PC should be sufficient to run this software. The program comes with prerecorded MJPEG Ethernet streams in *.PCAP-format and runs these streams through the TES 3D Surround View software.

13. The PC keyboard and mouse can be used to change the viewer position: predefined views are available on the number keys and “free-flight” is possible via mouse and (Shift-) arrow keys. Also, some of the 3D Surround View system parameters like the bowl-size and -shape as well as the stitching parameters can be adjusted. Do not hesitate to get in touch with TES (coordinates below) to get this Software. First steps in the Lab and on the vehicle Now that you have played with the software (and assuming it does more or less what you need) you might want to give it a try with real-life cameras, either in lab conditions or right away on the vehicle. The fastest way to do that is to get a “First Sensor Mobility Area View Kit – DC–AVK” (contents see above under “System Setup”) and the TES 3D Surround View Windows PC Software in Online-Mode (Evaluation Version). This kit allows lab setups as well as on-vehicle installation using a Windows Laptop. Up to 20m cable length (between cameras and switch) has been tested so far. Also here TES can assist you in purchasing this setup.

14. And now on the embedded hardware… Having tested the system under lab conditions and on the vehicle that target platform aspects now get focus (again). I.e. you want to see your setup running on the potential target hardware. At that point, you may want to take your Apalis iMX6 module with Ixora carrier board (or buy some from Toradex if you do not have those, yet) and get the TES 3D Surround View software for that target platform (prototype version). I.e. you get rid of the PC and use the Apalis/Ixora platform instead. The TES prototype software for that target platform has the same functionality as the evaluation version for PC and you can re-use the configurations that you have setup at the earlier stage now on the target.

15. Getting to a product For a product, you now have to customize the system to meet your specific application requirements. I.e. you use the Apalis iMX6 module as core and might re-design the carrier board based on the Ixora design data to meet your specific requirements regarding, interfaces, power supply, mechanics etc. In that step or even before you also may want to exchange the cameras with other types. Again, TES can assist you also in the hardware and system customization, development and (if wanted) manufacturing process.

16. Summary We have introduced an embedded 3D Surround View Turn-Key Solution and explained a process to get you familiar with this topic starting at zero cost. Following this process, your product development risk is significantly reduced because you can evaluate the system in a step-by-step approach and at the same time determine your application specific requirements. Finally, we have introduced the steps to come to a product based on this turn-key solution. Just get started!

17. Thank you