Remote Observing with the Keck Telescopes

1. Remote Observing with the Keck Telescopes Robert Kibrick, Director of Scientific Computing, University of California Observatories / Lick Observatory Science, Culture, and Education over Internet2 Networks, April 4, 2001

2. Overview of Presentation • Background • The Keck Telescopes • Mauna Kea Observatories • Telescope Scheduling • Modes of Observing • Remote observing with the Keck Telescopes • From Keck Headquarters in Waimea, Hawaii (32 km) • From Santa Cruz, California via Internet2 (3200 km) • Operational models and issues • Live videoconference with astronomers at Keck

3. The Keck Telescopes

4. The Keck Telescopes

5. Keck Telescope Facts • Twin, 10-meter optical/infrared telescopes • Largest telescopes of this type in the world • Construction funded by W. M. Keck Foundation • Observing time shared between 4 institutions • California Institute of Technology (Caltech) • University of California (UC) • National Aeronautics and Space Adminstration (NASA) • University of Hawaii (UH) • Located atop 4,200 meter summit of Mauna Kea

6. Mauna Kea summit on the island of Hawaii

7. Mauna Kea Summit • Dormant volcano • Premier astronomy site in N. Hemisphere • Above 90% of water vapor in atmosphere • Non-turbulent airflow over the summit • Sub-arcsecond atmospheric seeing • Home to many international observatories

8. Mauna Kea Optical Observatories • UH 0.6 meter • UH 2.2 meter • Gemini North • CFHT • NASA IRTF • Keck-2 • Keck-1 • Subaru

9. Largest Optical Telescopes in the Northern Hemisphere • Hubble Space Telescope Mirror is similar in size to UH 2.2-meter • Combined light gathering power of Mauna Kea telescopes is 50 times greater than HST

10. Common Facilities Operated by University of Hawaii • Dormitories atHale Pohaku • Altitude is 2,800m • All water must be trucked in • Acclimatization required before ascent to the 4,200m summit

11. Mauna Kea Observatories Serve An International Community • Argentina • Australia • Brazil • Canada • Chile • France • Japan • Netherlands • Taiwan • United Kingdom • United States

12. Assigning Telescope Time: Classical Scheduling • Committee evaluates observing proposals • Produce a 3 or 6 month telescope schedule • Approved proposals assigned a set of dates • Observing dates known many weeks in advance • Airline tickets can be purchased at lower rates • Astronomers can adjust course schedules • Proposal writers conduct their own observations • Can adjust observing program to weather conditions • Can alter program in case of unexpected discovery

13. Assigning Telescope Time:Queue Scheduling • Committee evaluates observing proposals • Approved proposals placed in a queue • Computer selects queue entries nightly: • Specific visibility or timing requirements • Best match of current sky conditions • Proposal priorities • Observing dates not known in advance • Proposal authors do not conduct observations

14. Queue scheduling works best if instruments can be changed quickly • Smaller, more recent telescopes, like the Subaru, have robotic mechanisms that permit instruments to be rapidly changed in the middle of the night. • This flexibility makes queue scheduling easier to implement.

15. Keck instruments must be changed manually and only during the day • Instrument size scales with telescope size. • Keck instruments are massive – some weigh nearly 8,200 kilograms. • Keck instruments can only safely be changed manually and during the daytime.

16. Keck Telescopes use Classical Scheduling • Kecks not designed for queue scheduling • Schedules cover a semester (6 months) • Approved proposals get 1 or more runs • Each run is between 0.5 to 4 nights long • Gaps between runs vary from days to months • Half of all runs are either 0.5 or 1 night long • Separate schedules for the two Kecks

17. Keck Telescope Schedule

18. Non-queue Scheduled Observing Modes • Local Observer • Controls observation from the telescope site • Remote Observer • Controls observation from a remote site • Service Observer • Observes in place of a remote observer • Remote observer submits detailed object list • Service observer conducts observation locally

19. Keck Telescope Observing Modes • Modes neither supported nor planned: • Service observing • Queue scheduled observing • Supported observing modes: • Local observing • Remote observing • Astronomers who are granted observing time conduct their own observations

20. From 1993 to 1995, all Keck observing was done at the summit Observers at the summit work from control rooms located adjacent to the telescope domes

21. Conducting observations involves coordinated effort by 3 groups • Telescope operator (observing assistant) • Responsible for telescope safety & operation • Keck employee; normally works at summit • Instrument scientist • Expert in operation of specific instruments • Keck employee; works at summit or Waimea • Observers • Select objects and conduct observations • Employed by Caltech, UC, NASA, UH, or other

22. Keck 2 Control Room at the Mauna Kea Summit Telescope operator, instrument scientist, and observers work side by side, each at their own computer.

23. Observing at the Mauna Kea summit is both difficult and risky • Oxygen is only 60% of that at sea level • Lack of oxygen reduces alertness • Observing efficiency significantly impaired • Altitude sickness afflicts some observers • Some are not even permitted on summit: • Pregnant women • Those with heart or lung problems

24. Initiative to support remote observing from Keck Headquarters • 1995: Remote control rooms built at Keck HQ • Initial tests via 1.5 Mbps (T1) link to the summit • 1996: Videoconferencing connects both sites • Remote observing with Keck 1 begins • 1997: >50% of Keck 1 observing done remotely • Link to the summit upgraded to 45 Mbps (DS3) • 1999: remote observing >90% for Keck 1 and 2 • 2000: remote observing is now the default mode

25. Keck 2 Remote Control Room at the Keck Headquarters in Waimea Observer and instrument scientist in Waimea use video conferencing system to interact with telescope operator at the summit

26. Videoconferencing has proved vital for remote observing from Waimea • Visual cues (body language) important! • Improved audio quality extremely valuable • A picture is often worth a thousand words • Chose dedicated versus PC-based units: • Original (1996) system was PictureTel 2000 • Upgrading to Polycom Viewstations

27. Keck 2 Remote Observing Room as seen from the Keck 2 summit Telescope operators at the summit converse with astronomer at Keck HQ in Waimea via the videoconferencing system.

28. The Remote Observing Facility at Keck Headquarters in Waimea • Elevation of Waimea is 800 meters • Adequate oxygen for alertness • Waimea is 32 km NW of Mauna Kea • 45 Mbps fiber optic link connects 2 sites • A remote control room for each telescope • Videoconferencing for each telescope • On-site dormitories for daytime sleeping

29. The Keck Headquarters in Waimea Most Keck technical staff live and work in Waimea. Allows direct contact between observers and staff. Visiting Scientist’s Quarters (VSQ) located in same complex.

30. Limitations of Remote Observing from Keck HQ in Waimea Most Keck observers live on the mainland. Mainland observers fly > 3,200 km to get to Waimea Collective direct travel costs exceed $400,000 U.S. / year

31. Remote Observing from Waimea is not cost effective for short runs • Round trip travel time is 2 days • Travel costs > $1,000 U.S. per observer • About 50% of runs are for 1 night or less • Cost / run is very high for such short runs • Such costs limit student participation

32. Motivations for Remote Observing from the U.S. Mainland • Travel time and costs greatly reduced • Travel restrictions accommodated • Sinus infections and ruptured ear drums • Late stages of pregnancy • Increased options for: • Student participation in observing runs • Large observing teams with small budgets • Capability for remote engineering support

33. Fast and reliable network needed for mainland remote observing • 1997: 1.5 Mbps Hawaii -> Oahu -> mainland • 1998: 10 Mbps from Oahu to mainland • 1999: First phase of Internet-2 upgrades: • 45 Mbps commodity link Oahu -> mainland • 45 Mbps Internet-2 link Oahu -> mainland • 2000: Second phase upgrade: • 35 Mbps Internet-2 link from Hawaii -> Oahu • Now 35 Mbps peak from Mauna Kea to mainland

34. Internet-2 links

35. Mainland remote observing goals and implementation strategy • Goals: • Target mainland facility to short duration runs • Avoid duplicating expensive Waimea resources • Avoid overloading Waimea support staff • Strategy: • No mainland dormitories; observers sleep at home • Access existing Waimea support staff remotely • Restrict mainland facility to experienced • Restrict to mature, fully-debugged instruments

36. Mainland remote observing facility is an extension of Keck HQ facility • Only modest hardware investment needed: • Workstations for mainland remote observers • Network-based videoconferencing system • Routers and firewalls • Backup power (UPS) – especially in California!!! • Backup network path to Mauna Kea and Waimea • Avoids expensive duplication of resources • Share existing resources wherever possible • Internet-2 link to the mainland • Keck support staff and operational software

37. Keck software is accessed the same regardless of observer’s location • The control computers at the summit: • Each telescope and instrument has its own computer • All operational software runs only on these computers • All observing data written to directly-attached disks • Users access data disks remotely via NFS or ssh/scp • The display workstations • Telescopes and instruments controlled via X GUIs • All users access these X GUIs via remote displays • X Client software runs on summit control computers • Displays to X server on remote display workstation

38. Why did we choose this approach? • Operational Simplicity • Operational control software runs only at the summit • All users run identical software on same computer • Simplifies management between independent sites • Allowed us to focus on commonality • Different sites / teams developed instrument software • Large variety of languages and protocols were used • BUT: all instruments used X-based GUIs

39. Focus effort on X standardization and optimization over long links • Maintain consistent X environment between sites • Optimize X performance between sites • Eliminates need to maintain: • Diverse instrument software at multiple sites • Diverse telescope software at multiple sites • Coordinate users accounts at multiple sites • Fewer protocols for firewalls to manage

40. Accessing Keck software and data from Keck HQ in Waimea Telescope operator uses display workstation at summit. Instrument scientist and observers use display workstations in Waimea.

41. Accessing Keck software and data from the mainland Telescope operator uses display workstation at summit. Instrument scientist uses display in Waimea Observers use display on mainland

42. Remote observing differences: Waimea versus the mainland • System Management: • Keck summit and HQ share a common domain • Mainland sites are autonomous • Remote File Access: • Observers at Keck HQ access summit data via NFS • Observers on mainland access data via ssh/scp • Propagation Delays: • Summit to Waimea round trip time is about 1 ms. • Summit to mainland round trip time is about 100 ms.

43. Increased propagation delay to mainland presents challenges • Initial painting of windows is much slower • But once created, window updates fast enough • All Keck applications display to Waimea OK • A few applications display too slowly to mainland • System and application tuning very important • TCP window-size parameter (Web100 Initiative) • X server memory and backing store • Minimize operations requiring round trip transactions

44. Tradeoffs from this approach to remote observing • Disadvantages: • X protocol does not make optimal use of bandwidth • Long propagation delays require considerable tuning • Advantages: • Minimizes staffing requirements at mainland sites • Only “vanilla” hardware and software needed there • Simplifies sparing and swapping of equipment • Simplifies system maintenance at mainland sites

45. End-to-end reliability is critical to successful remote operation • Keck Telescope time is valued at $1 per second • Each observer gets only a few nights each year • Observers won’t use facility if not reliable • What happens if network link to mainland fails? • Path from Mauna Kea to mainland is long & complex • At least 14 hops crossing 7 different network domains • While outages are rare, consequences are severe • Even brief outages cause session collapse & panic • Observing time loss can extend beyond outage

46. Mitigation plan: install end-to-end ISDN-based fallback path • Install ISDN lines and routers at: • Each mainland remote observing site • Keck 1 and Keck 2 control rooms • Fail-over and fallback are rapid and automatic • Toll charges incurred only during network outage • Lower ISDN bandwidth reduces efficiency, but: • Observer retains control of observations • Sessions remain connected and restarts avoided • Prevents observer panic

47. Summary of ISDN-based fallback path • Install 3 ISDN (6 B channels) between sites • Install Cisco 2600-series routers at each end • Dual auto-sensing Ethernet interfaces • Quad BRI interfaces • Inverse multiplexing • Dial-on-demand (bandwidth-on-demand) • Caller ID (reject connections from unrecognized callers) • Multilink PPP with CHAP authentication • Uses GRE tunnels and OSPF routing • No manual intervention needed at either end

48. Summary of progress to date • Prototype mainland facility assembled at UCSC • Multipoint videoconferencing to summit and HQ • Efficiently displays telescope & instrument status • Two Keck instruments operated remotely: • High Resolution Spectrometer (HIRES) on Keck 1 • Echellette Spectrogram & Imager (ESI) on Keck 2 • Primary use has been for remote engineering • On-sky remote observing performed with ESI • Efficient, automated transfer of image files

49. Future Plans • Conduct ongoing trials using prototype at UCSC • Work out operational details with Keck staff • Add capability to operate additional instruments • Aim for fully operational status by late 2001 • Extend to other sites once debugged at UCSC

50. Items Currently in Progress • Real time display of guide camera images • Installation of ISDN lines at the Keck summit • Installation of ISDN routers at the Keck summit • Adjustments to telescope scheduling procedures • Firewall issues at summit and mainland sites