1 / 16

Outline

Ground Based Observatories (GBO) Engineering Peer Review Oct. 17, 2003 Data Retrieval and Software Configuration Brian Jackel University of Calgary. Outline. Imager data processing transmission and storage Site computer timing monitoring security. Imager on-site data flow. Network

reuben
Download Presentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ground Based Observatories (GBO)Engineering Peer ReviewOct. 17, 2003Data Retrieval and Software ConfigurationBrian JackelUniversity of Calgary

  2. Outline • Imager data • processing • transmission and storage • Site computer • timing • monitoring • security

  3. Imager on-site data flow Network gateway On-site storage network transport physical transport

  4. Imager data center network transport Test Receiver Receiver (warm spare) Public Storage physical transport Storage http, scp, rsync.... Data products Data

  5. Imager on-site data flow • imagerd always running • acquisition from camera when sun-angle > 108° • prepend site information site_info.conf • store as ASCII PGM /tmp/recent_image.pgm • create thumbnail • convert image to PNG, copy to /data/imager/ tree /tmp/recent_image.pgm P2 #"Site code" CALGU #"Site name" Calgary University #"Site location" 51.1139N 114.02W 1084m #"Site notes" University of Calgary Physics #"Imager code" THEMIS01 #"Exposure start time" 2003-09-27 04:38:45 UTC #"Exposure finish time" 2003-09-27 04:38:47 UTC #"Exposure duration" 1000 ms #"Exposure type" VIS #"Image size" 376 290 #"CCD binning" 2 1 #"CCD offset" 0 0 #"CCD DAC" 16 #"CCD device" /dev/ccdA 376 290 65535 3124 2900 2929 3018 3087 2992 2942 3166 2954 3001 2906 2910 2998 3066 2909 3041 3017 3015 2972 3214 3082 2955 3028 2974 2966 2957 3041 2939 3107 ....

  6. Image to thumbnail If an ASI frame looked like this: Then the a coarsely binned map of aurora will look like this: y lat x lon

  7. Thumbnail to image If the lat/lon auroral map looked like this: Then the ASI frame would look like this: Make a “thumbnail bins” image file where each pixel value provides a reverse mapping to the corresponding thumbnail bin. Use as quick lookup table. Needs to be calculated each time a camera is installed or moved.

  8. Image thumbnails /usr/local/etc/thumbnail_bins.pgm P5 # ATHA THEMIS00 VIS # NBINS 20 20 # MLAT 56-61,0.5 # MLON 129-135,1.0 # binning created 2003-10-03 14:15:15 376 290 65535 ...sequence of 16-bit binary values... • thumbnaildalways running • wait for change in timestamp of/tmp/recent_image.pgm • runpgmtothumbwhich usesthumbnail_bins.pgm to average 16-bit pixel values in each bin, convert to 8-bits, output as binary PGM to /tmp/recent_thumbnail.pgm • copyto/data/imager/tree • compress withgzipandtransmit over network withudp_sendfile /tmp/recent_thumbnail.pgm P5 # ATHA THEMIS00 VIS # NBINS 20 20 # MLAT 55-65,0.5 # MLON 129-149,1.0 # binning created 2003-10-03 14:15:15 20 20 255 ...sequence of 8-bit binary values...

  9. Orientation from Stars instrument_name Themis00 site_name Athabasca site_code ATHA geodetic_latitude 54.72 geodetic_longitude 246.72 geodetic_altitude 250.0 camera_yaw 58.0 camera_tilt 0.5 camera_pitch 0 optics_a1 1.53 optics_a2 0.0 optics_a3 0.0 optics_b1 0.0 optics_b2 0.0 ccd_nx 290 ccd_ny 290 ccd_x0 146.0 ccd_y0 155.0 ccd_xflip 1 ccd_yflip -1

  10. Data directories and filenames • Data directory tree: • /data /mag/... /status/<year>/<month>/<day> /imager/<year>/<month>/<day>/<site>_<instrument>/ut<hour> • eg. /imager/2003/10/17/ATHA_THEMIS00/ut02 • Image filename convention: • <date>_<time>_<site>_<instrument>_<mode>_<exposure>.png • eg. 20031017_024205_ATHA_THEMIS00_VIS_1000ms.png At 5-second cadence there will be 720 images/hour

  11. /usr/local/etc/site_info.conf #"Site code" ATHA #"Site name" Athabasca Geophysical Observatory #"Site location" 54.72N 246.72E 676m #"Site notes" Athabasca University in northern Alberta Site-specific configuration • Minimize software configuration differences between site computers • physical location • network configuration • image thumbnail map • local log files • ??

  12. Computer site timing • computer clocks typically not reliable • network time synchronization (NTP) over satellite may require excessive bandwidth, not reliable • GPS provides 1 pulse-per-second with 50ns accuracy, UTC time string for NTP reference • tests of Trimble Palisade unit indicate offsets on the order of 20ms using RS-232 ASCII signal • software conflicts can prevent NTP synchronization /usr/sbin/ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== LOCAL(0) LOCAL(0) 10 l 29 64 377 0.000 0.000 0.015 -wham.phys.ucalg CLOCK.UREGINA.C 2 u 55 64 377 0.343 20.253 0.057 xdancal.phys.uca CLOCK.UREGINA.C 2 u 8 64 377 0.569 76.535 1.921 +fsa.cpsc.ucalga subitaneous.cps 2 u 64 64 377 2.301 20.129 0.050 +time.nrc.ca tac.nrc.ca 2 u 28 64 377 96.307 18.981 0.265 *GPS_PALISADE(1) .GPS. 0 l 27 32 377 0.000 -0.482 0.071

  13. Site computer monitoring • CPU and motherboard temperatures, fan speeds • hard drive temperature, error rates • clock status (NTP) • network status • power status (UPS) • free disk space, memory • web-cam? • ??

  14. Data synchronization • Every day, each site computer will • calculate checksums for previous day of thumbnails • copy data to external disk (if present) • if less than 5% free disk space, lossy compress or delete oldest data • Every day, a central computer will • download checksums from remote sites • identify intervals of missing or compromised thumbnails, pull from sites

  15. Site computer security • Any system on the internet will eventually be attacked. Plan for this now, or waste time reacting later. • linux software fully patched before deployment • accounts: imager (UCalgary, UCB), mag (UCLA, UAlberta), rootfor emergency/maintenance only • firewall: access only from specific subnets (ie. phys.ucalgary.ca, ssl.berkeley.edu, phys.ualberta.ca, ...) • firewall: access only from specific ports (ie. ssh, rsync, ntp, ...) • login attempts logged locally and remotely, key configuration files checked periodically

  16. Site software changes • don't (unless absolutely necessary) • document reasons for and specific details of change • run on test machine for as long as possible • push to one active site after making snapshot of complete current state • push to other active sites, again after snapshotting • burn CDs with new basic state, keep with copies of previous states

More Related