Don Appleman, Visiting Research Programmer , Imaging Technology Group

1. ITG Forum Constructing a Research Grade Interactive Remote Microscopy Application: Goals, Challenges, and Results Don Appleman, Visiting Research Programmer, Imaging Technology Group April 1, 2003

2. ITG Mission Mission of the Imaging Technology Group • Provide state-of-the-art imaging facilities for Institute researchers. • Develop advanced imaging technologies with emphasis on projects in remote and automated instrument control.

3. Remote Instrumentation Remote access to imaging instrumentation Why? • Unique and/or expensive instrumentation • Distributed expertise • Outreach; Education; Training of professional researchers and engineers

4. What is Bugscope? What is Bugscope? • Web-based access to an Environmental Scanning Electron Microscope (ESEM) for K-12 classrooms to image “bugs”. • Schools in 33 states have used the ESEM over the Internet in more than 120 sessions using the current software.

5. Current Technology Initial Implementation • Separate Controller & Observer software • Chat Window for conversing with microscopist • Java applets or CGI interface to microscope server software • Still images only

6. Current Technology

7. Current Technology Drawbacks • Still images only • Time to acquire image is long This results in the following problems … • Difficulty navigating across the sample • Difficulty focusing • Loss of concentration on the task at hand

8. Current Technology Current Project • The current project is a remote instrumentation project funded by NASA. • NASA is interested in the remote operation of complex scientific equipment over long distances in real time. • The project goal is to enhance the Bugscope software, known as the Interactive Remote Microscopy Application (IRMA), to the point where it qualifies as a research-grade interface.

9. Future Technology Approach — Improved Input & Output • Input — Sufficient control of the microscope + • Output — True-to-life viewing experience = Goals • Remote and collaborative research among isolated researchers • Training new users to operate the ESEM • Teaching classes ranging from nanoscience to biology

10. Future Technology Proposal • Add streaming video to Bugscope

11. Future Technology Proposal • Add streaming video to Bugscope Challenges • Bandwidth requirements • Image quality • Image latency • Image size

12. Streaming Video For a given bandwidth capacity, pick any two • Image quality Fidelity of the reconstructed image to the original source • Image latency Delay in real-time between acquisition of the image and display of the image to the remote user • Image size The size of the image displayed to the remote user

13. Streaming Video Videocast options • Broadcast • Video stream sent to everyone on a limited network. • Server bandwidth equivalent to a single stream. • Creates lots of network traffic. • Not compatible with Internet delivery to arbitrary users. • Unicast • Dedicated video stream sent to a single, known client. • Serves a single client.

14. Streaming Video Videocast options, cont’d • Multi-unicast • Multiple dedicated video streams sent to multiple clients. • Server bandwidth multiplied by number of simultaneous users. • Client bandwidth multiplied by number of simultaneous users on given Internet feed (classroom full of students). • Multicast • Video stream sent to abstract address, replicated by routers. • Server bandwidth equivalent to a single stream. • Client bandwidth equivalent to a single stream for Internet feed. • Requires properly configured, compatible routers.

15. Streaming Video Videocast solution Hybrid Video Server • Initiate multicast • Listen for client connections • Inform clients of the active multicast address • If the client reports that it cannot see the multicast • Start a dedicated unicast to client address • Inform client of unicast address:port • Monitor client status

16. Streaming Video Project-specific Advantages • No audio • Monochromatic source The solution • H263 video-conferencing video format • Video server written in Java (using Java Media Framework) • Java applet video client (adjunct to extant software)

17. Streaming Video Project-specific Results, low bandwidth • Video stream bandwidth ~100Kbps • Image quality acceptable (some quality lost, still image needed for actual study); good enough for fine focus control using current interface • Image latency less than or equal to 500 ms • Image size 352x288 • Video frame rate ~15 frames per second (variable bit rate)

18. Streaming Video Project-specific Results, high bandwidth • Video stream bandwidth ~3Mbps • Image quality very good (JPEG quality 0.8, still image usually not needed for actual study) • Image latency less than or equal to 500 ms • Image size 640x480 • Video frame rate ~15 frames per second (variable bit rate)

19. Beyond Video A new, more complete and more natural user interface. Expose the majority of the controls accessible when using the ESEM directly.

20. Beyond Video New controls • Beam on/off, kV, and spot size • Scan mode (Slow scan 1, 2, 3, or TV mode) • Filters (Live, Integrate 1, Average 4) • Detectors (Secondary electron, Backscatter electron, CCD camera) Improved controls • New Navigation, Focus, and Measurement Tools

21. Beyond Video Not all controls can be included • Cannot change specimens • Cannot activate vacuum • Cannot operate manual tilt mechanism An operator is required to prepare and mount the specimen, and to activate the vacuum.

22. Navigation Tool ESEM Navigation Click anywhere relative to the green overlay circle, and the stage will move in that direction. In “TV” mode, click & hold moves smoothly in given direction.

23. Navigation Tool • Bugscope Navigation • Control+click on Observer to center image on selected location • Controller stage control • Controller presets

24. Navigation Tool Proposed Navigation • Click & hold over video window to move as if in “TV” mode on ESEM • Keyboard or other control to move by ½ screen in any direction • Control+click on video display to center image on selected location

25. Focus Tool ESEM Focus Click & drag right mouse button left/right to increase/decrease focus distance. Control+click & drag right mouse button left/right or up/down to adjust stigmator x,y.

26. Focus Tool • Bugscope Focus • Coarse control +/- to adjust focus by 1mm • Fine control to adjust by 0.1mm to 0.9mm • ESEM Auto Focus (slow, can be unreliable)

27. Focus Tool Proposed Focus (same as ESEM focus) • Using video window, click & drag right mouse button left/right to increase/decrease focus distance. • Using video window, control+click & drag right mouse button left/right or up/down to adjust stigmator x,y. • May also use trio of images, side by side, each showing a different focal depth that the user can select.

28. User Base Varying user base • Educational Outreach (Bugscope) • Bandwidth limited (usually) • Little or no experience operating a real ESEM • Not fully accountable • Chat with professional required • Professional Researchers • Lots of bandwidth (usually) • May have experience operating a real ESEM • Higher level of accountability • Professional assistance unneeded

29. User Base Varying user base, solution Configurable control set • For educational outreach, eliminate controls that are confusing, hard to use, or which might damage the equipment. Configurable video server • Configure to use available bandwidth

30. Conclusions Conclusions: • With the addition of streaming video to Bugscope, the issue of realism for the output of the IRMA software has been addressed. • Once we complete the enhancements to the input interface of the Controller, IRMA will be a true, research-ready option for usage of the ESEM. • With the requirement that an operator be present to prepare and mount the specimens, and to operate the vacuum controls, IRMA will remain limited. • As a proof-of-concept of remote instrumentation capabilities, the completed project is a success.

31. References The Bugscope web site http://bugscope.beckman.uiuc.edu Technical reports http://bugscope.beckman.uiuc.edu/publications/index.htm