NRAO VLA Archive Survey

National Radio Astronomy Observatory Jared Crossley Loránt Sjouwerman Edward Fomalont Nicole Radziwill 2008-06-25 SPIE: Observatory Operations: Strategies, Processes, and Systems NRAO VLA Archive Survey

(NVAS) • Survey of the VLA Archive generated by an automated data pipeline. • Under development since 2006. • Data products include: images, calibrated data, diagnostics, logs, … • Project ongoing, but data is already available on the Web. • http://archive.nrao.edu/ NRAO VLA Archive Survey

Very Large Array • Synthesis array of 27 antennas • Located in New Mexico, USA • Operational for 28+ years • Maximum baseline: 0.3 to 35 km • Frequencies: 0.07 to 50 GHz • Produces over 150 papers per year • Outputs “visibility” measurements

“Typical” VLA Data Reduction • Project scientists are responsible for all parts of the data reduction process. • Download VLA raw data from the VLA Archive. • Making images requires: • Careful editing and calibration • Fast Fourier transform processing • Deconvolution methods • To make data more easily accessible we have created the VLA Archive Survey

Present State of Survey

Sky Coverage

1. Acquire 2. Process 3. Archive 4. Validate NVAS Pipeline System • Makes use of preexisting components: • Raw data archive and interface • AIPS synthesis imaging software package • Perl and shell scripts connect components to form the pipeline. • NVAS data is incorporated in the raw data archive. One interface provides access to raw and processed data.

NVAS Pipeline Stages Acquire Data 1. Acquire • Perl script automates communication with Archive server by HTTP requests. • Input start and stop dates for processing • Script queries the server • Algorithm selects one or more data files • Download to local disk • Data is loaded into AIPS, split by frequency, and prepared for processing

NVAS Pipeline Stages Data Processing 1. Acquire • AIPS “runfile” (script) performs standard synthesis imaging steps (VLARUN) • Output data products: • Images • Calibrated data • Diagnostic plots • Log files • Metadata for Web interface 2. Process

Example image and diagnostic plots: Crab Nebula at 5 GHz

NVAS Pipeline Stages Archiving 1. Acquire • Shell and Perl scripts • Move data products to archive server • Generate HTML for survey Web interface 2. Process 3. Archive

NVAS Pipeline Stages Validation 1. Acquire • Human-interactive process • A private Web interface is used to remove low quality data. • All NVAS data is reviewed by a human. • Less than 5% of images have been removed during data validation. 2. Process 3. Archive 4. Validate

Pipeline System Performance • Only data validation requires human interaction. • All other pipeline stages are limited by computer and network data rates. • Data processing requires the most time; high-resolution imaging may take many hours • Survey data is 1.4 TB and growing

Data Vault: 7019-47 Access the survey from http://archive.nrao.edu/

Future Development • Finish imaging of VLA continuum data At the same time, continue algorithm upgrades: • Self-calibration • Better data editing • Mosaic imaging • Composite imaging of long-timescale projects • Apply new algorithms to all survey data • Process spectral line data Expect to finish these stages end of 2010.

NRAO VLA Archive Survey is… Summary • Automated survey of the full VLA archive • Makes VLA data more accessible to all astronomers • Provides reference images, calibrated data, diagnostics, logs, … • Ongoing survey, expected completion in 2010 • Currently available at: http://archive.nrao.edu/

NRAO VLA Archive Survey