170 likes | 274 Views
NIR Transient Surveys. Nicholas Cross WFAU, Edinburgh Nigel Hambly , Mike Read, Ross Collins, Eckhard Sutorius , Rob Blake, Mark Holliman. NIR Variability Science Drivers. NIR, smaller detectors, higher backgrounds and more expensive detectors than optical
E N D
NIR Transient Surveys Nicholas Cross WFAU, Edinburgh Nigel Hambly, Mike Read, Ross Collins, EckhardSutorius, Rob Blake, Mark Holliman
NIR Variability Science Drivers • NIR, smaller detectors, higher backgrounds and more expensive detectors than optical • Only do multi-epoch work where it is not practical for optical detectors • Looking through the dense dusty regions of the MW to the far side • Young Stellar Objects in star-forming regions • Low mass stars / brown dwarfs • High z galaxies / Snae • Can get better RR Lyrae / Cepheid distances in NIR IAU S285, Oxford
NIR Variability Surveys • UKIRT WFCAM • UKIDSS – DXS/UDS (Deep surveys, multi-epoch), • WFCAM Transit Survey, • Calibration/Standard Stars, • Surveys of YSOs in Orion/Ophiuchus • VISTA • VISTA Variables in Via-Lactea (VVV), (RR Lyrae, Cepheids) • VISTA Magellanic Cloud (VMC), (RR Lyrae, Cepheids) • VIDEO (Deep Extragalactic – SNae) IAU S285, Oxford
WFCAM • 3.8 m UKIRT telescope on Mauna Kea. • 4 2k x 2k Rockwell Hawaii 2 detectors. • Spaced 94% apart. • 0.4” pixels. • 13.65’ across each side. • 60% of time on UKIRT in 2005b • 100% for 2009a IAU S285, Oxford
VIRCAM • 4.1m VISTA telescope at Cerro Paranel. • 16 2k x 2k Raytheon VIRGO detectors • Spaced 90% in x and 42.5% in y. • 0.34” pixels • Tile is 1.5° • VIRCAM has 100% of time. • > 3 times area WFCAM • 2 * QE IAU S285, Oxford
VISTA Public Surveys IAU S285, Oxford
VISTA Variables in Via-Lactea (VVV) • Very high density ~106 sources / sq. deg. • Issues with deblending • 500 sq. deg • ~100 epochs (currently ~10) • ~ few 1010 detections IAU S285, Oxford
Processing of WFCAM and VISTA data • VDFS: VISTA Data Flow System (System for processing of UKIRT WFCAM and VISTA data. • CASU (Cambridge): Data reduction, processing of observing blocks, photometric and astrometric calibration • WFAU (Edinburgh): Archive, processing of multiple observing blocks – deep stacks, multi-band tables, links to external tables, MULTI-EPOCH. • For VISTA, data goes through ESO and final products go to ESO too. IAU S285, Oxford
Constraints from VDFS • >=6 week time lag before data at WFAU • Data needs to be transferred to Cambridge (with VISTA this includes disk drive to Garching and then to Cambridge) • Accurate photometric calibration (including scattered light corrections uses 1 month of data. • VoEvent alerts are too late from WFAU • Reprocessing of OB data requires retransfer between CASU and WFAU and reingest of data at WFAU. • Detection tables are used by many curation processes – reingestion into these slows later stages. IAU S285, Oxford
Stages of multi-epoch processing • Stack epochs to create deep images and extract catalogues • Create master list (Source table) from band-merged catalogues from deep images. • Recalibrate each epoch image compared to the deep image in that filter and pointing. • Create table linking sources to each observation • Calculate the noise properties of each pointing and filter • Calculate astrometric and photometric statistics for each source. IAU S285, Oxford
Analysing Variables • Calculate mean, rms of magnitudes. • Bin in magnitude and calculate clipped median • Fit empirical noise model • (m)=a+b10-0.4m+c10-0.8m • Classify as variable or non-variable IAU S285, Oxford
Archival Databases • Curation of WFCAM and VISTA data occurs in a RDBMS using Microsoft SQL Server. • Dynamic database, updated with new data, improved calibrations and reprocessed data when necessary. • Static releases to the science teams and world for science purposes. • Curation controlled by comparing current state of DB with requirements IAU S285, Oxford
Programme Requirements • Pointing, filter and table requirements are setup by grouping the metadata and using specifications for each survey. • Schema updated if necessary • Stack / tile products made for a particular release number • Source table created for particular pointings • Each stage of multi-epoch processing checks the whether the previous table has changed in that pointing – higher curation event ID. IAU S285, Oxford
VISTA tiles • Most surveys require tiles to reach expected depth, and tiles are standard ESO product. • PSF and sky vary on short time scales < integration time • Images filtered to remove large spatial variations (>30”) • Tile catalogues are inferior to pawprints: • Not as accurate astrometry • Do not deal with saturation correctly • Extended (>30”) sources are missing or have incorrect photometry • Catalogues from tiles and pawprints • Need to be able to compare – multiple layers and linking tables. IAU S285, Oxford
Problems / bottlenecks /solutions • Reprocessing of OB data. • 1st year of VISTA – 2 sets of full reprocessing • Ingesting new data while curating later products • Put VVV on separate server and synchronise metadata tables • BUT foreign key constraints to vvvDetection cause major holdups if metadata is deleted. • Split vvvDetection into semesters / months so new data can be ingested into new semester. • Has not been implemented yet • Users want to use both tile and pawprint detections • Produce linking tables • BUT some queries that join these can join several tens of tables and SQL does not handle these joins well. • Enhancements to user interface allow users to save intermediate results IAU S285, Oxford
Problems / bottlenecks /solutions • Checking non-detections of sources • Using half-space method of Budavari, major improvement • Dealing with very long processing times of VVV • Break curation into chunks with software testing to see what has already been done • Make sure memory never exceeds ~40% • BUT this adds additional overheads at beginning of each run • Variability table curation is dominated by DB reads (85% for VVV) • Use Query Analyser and other tools to optimise queries [OPTION (MAXDOP 1)], adding removing indexes. • Split detection tables into parts? • I/O limited between servers and disks • SQL Server “cluster” linked by infiniband 10Gbs-1 IAU S285, Oxford
Other issues • Classification • DB has simple classification (variable or not) and some other statistical quantities. VVV will have ~106 variables • Chilean teams working on NIR templates for different types of variables • Trend analysis (IstvanDekany) • Accuracy • VSA/WSA, simple ZP recalibration – rms ~0.005mag • Good enough for most variables • Planetary Transits require (prefer) ~0.001 mag. • Confusion • Difference Imaging Analysis (EamonnKerins), will probably be applied to densest 40 sq. deg of VVV bulge. IAU S285, Oxford