Calibration

1. Michael Bietenholz Calibration Based on a lecture by George Moellenbrock (NRAO) at the NRAO Synthesis Imaging Workshop

2. Synopsis • Why calibration and editing? • Editing and RFI • Idealistic formalism → Realistic practice • Practical Calibration • Baseline- and Antenna-based Calibration • Intensity Calibration Example • Full Polarization Generalization • A Dictionary of Calibration Effects • Calibration Heuristics • New Calibration Challenges • Summary

3. Why Calibration and Editing? • Synthesis radio telescopes, though well-designed, are not perfect (e.g., surface accuracy, receiver noise, polarization purity, stability, etc.) • Need to accommodate deliberate engineering (e.g., frequency conversion, digital electronics, filter bandpass, etc.) • Passage of radio signal through the Earth’s atmosphere • Hardware or control software occasionally fails or behaves unpredictably • Scheduling/observation errors sometimes occur (e.g., wrong source positions) • Radio Frequency Interference (RFI) Determining instrumental properties (calibration) is a prerequisite to determining radio source properties

4. Calibration Strategy • Observe calibrator sources in addition to our program sources • These are sources with a known location and known properties, usually point sources (or nearly so) • Ideally, they are nearby on the sky to our target source • By examining the visibility measurements for the calibrator sources, where we know what they should be, we can estimate our instrumental properties, often called the calibration • We can then use these estimates of the instrumental properties to calibrate the visibility data for the program source • In general the instrumental properties vary with time, with frequency and with position on the sky • One usually uses different calibrator sources to obtain different parts of the calibration (flux density scale, polarization etc, etc), trying to separate out those aspects which change on different timescales (generally: instrumental – long timescales; atmosphere – short timescales)

5. What Does the Raw Data Look Like? Flux Density Calibrator – e.g., 3C286 AIPS Task: UVPLT Visibility Amplitude Program source Phase calibrator Time Time

6. Calibration and Editing • Calibration and editing (flagging) are inter-dependent. If we derive calibration from visibilities, we want to edit out corrupted visibilities before obtaining calibration • But: editing data is much easier when its already well calibrated • Integration time – the time interval used to dump the correlator, typically 1 – 10 secs • Scan – One continuous observation of one source, typically 1 to 30 minutes Terminology

7. What Does the Raw Data Look Like? Flux Density Calibrator – e.g., 3C286 AIPS Task: UVPLT Visibility Amplitude Program source Bad data, to be flagged Phase calibrator Time Time

8. What Does the Raw Data Look Like? AIPS Task: UVPLT

9. AIPSTVFLG Color: visibility amplitude in this example. Can be phase or other quantities Time  Baseline 

10. Don’t Edit Too Much Rule 1) You should examine your data to see if there is anything that needs to be edited out. If your data is good, there may be nothing to edit out, but you won’t know till you look! Rule 2) Try to edit by antenna, not by baseline. The vast majority of problems are antenna-based, so if baseline ant 1 – ant 2 is bad, try and figure out whether its ant 1 or ant 2 which has the problem and then flag the antenna. Caveat: RFI is generally baseline-based. Rule 3) Don’t edit out data which is just poorly calibrated – fix the calibration instead. Rule 4) Don’t be afraid of noise – much of our visibility data, especially on weak sources, looks very much like pure noise. Don’t throw it out – the signal you want is buried in that noise. Rule 5) Don’t edit too much! • The goal is to remove data which is obviously bad. Generally, if you are editing out more than 10% of your data, you are probably editing too much. Rule 6) Remember your program source. If e.g., an antenna is bad for two calibrator scans, its probably bad for the intervening program source scan, and should be edited out.

11. Radio Frequency Interference • Has always been a problem (Grote Reber, 1944, in total power)!

12. Radio Frequency Interference (cont) • Growth of telecom industry threatening radio astronomy!

13. Radio Frequency Interference • RFI originates from man-made signals generated in the antenna electronics or by external sources (e.g., satellites, cell-phones, radio and TV stations, automobile ignitions, microwave ovens, computers and other electronic devices, etc.) • Adds to total noise power in all observations, thus decreasing the fraction of desired natural signal passed to the correlator, thereby reducing sensitivity and possibly driving electronics into non-linear regimes • Can correlate between antennas if of common origin and baseline short enough (insufficient decorrelation via geometry compensation), thereby obscuring natural emission in spectral line observations • Some RFI is generated by the instruments themselves (Local oscillators, high-speed digital electronics, power lines). Careful design can minimize such internal RFI. • Least predictable, least controllable threat to a radio astronomy observation.

14. Radio Frequency Interference • RFI Mitigation • Careful electronics design in antennas, including filters, shielding • High-dynamic range digital sampling • Observatories world-wide lobbying for spectrum management • Choose interference-free frequencies: but try to find 50 MHz (1 GHz) of clean spectrum in the VLA (EVLA) 1.6 GHz band! • Observe continuum experiments in spectral-line modes so affected channels can be edited • Various off-line mitigation techniques under study • E.g., correlated RFI power that originates in the frame of the array appears at celestial pole (also stationary in array frame) in image domain…

15. Calibration: What Is Delivered by a Synthesis Array? • An enormouslist of complex numbers (visibility data set)! • E.g., the EVLA: • At each timestamp (~1s intervals): 351 baselines (+ 27 auto-correlations) • For each baseline: 1-64 Spectral Windows (“subbands” or “IFs”) • For each spectral window: tens to thousands of channels • For each channel: 1, 2, or 4 complex correlations • RR or LL or (RR,LL), or (RR,RL,LR,LL) • With each correlation, a weight value • Meta-info: Coordinates, antenna, field, frequency label info • Ntotal = Nt x Nbl x Nspw x Nchan x Ncorr visibilities • EVLA: ~1300000 x Nspw x Nchan x Ncorr vis/hour (10s to 100s of GB per observation) • MeerKAT: ~8X more baselines than EVLA!

16. Calibrator Sources • Ideally – they would be very strong, completely point-like sources which did not vary in time • In practice such sources do not exist. Only a few sources have reasonably stable flux densities, and they are usually not very compact. • Most point-like sources, on the other hand, are variable with time (timescales from days to weeks) • Typical strategy is to use one of the few stable sources as a flux-density calibrator, observed once or twice in the observing run, and a point-like source near the program source as a phase calibrator, which is observed more frequently.

17. AIPS Calibration Philosophy • “Keep the data” • Original visibility data is not altered • Calibration is stored in tables, which can be applied to print out or plot or image the visibilities • Different steps go into different tables • Easy to undo • Need to store only one copy of the visibility data set (big file), but can have many versions of the calibration tables (small files)

18. AIPS Calibration Tables • Visibility data file contains the visibility measurements (big file). Associated with it are various tables which contain other information which might be needed: here are some of the tables used during calibration: • AN table – Antenna table, lists antenna properties and names • NX table – Index table, start and end times of scans • SU table – Source table, source names and properties (e.g., flux density if known) • FQ table – frequency structure. Frequencies of different IFs relative to the header frequency • FG table – flagged (edited) data, marks bad visibilities • SN table – “solution table” , contains solutions for complex gains as a function of time and antenna • CL table – complex gains as a function of time and antenna interpolated to a regular grid of times, this is the table that is used to actually calibrate the visibilities different tables • BP table – bandpass response, complex gain as a function of frequency and antenna

19. From Idealistic to Realistic • Formally, we wish to use our interferometer to obtain the visibility function: • ….which we intend to invert to obtain an image of the sky: • V(u,v) set the amplitude and phase of 2D sinusoids that add up to an image of the sky • How do we measure V(u,v)?

20. From Idealistic to Realistic • In practice, we correlate (multiply & average) the electric field (voltage) samples, xi & xj, received at pairs of telescopes (i, j ) and processed through the observing system: • xi& xjare delay-compensated for a specific point on the sky • Averaging duration = integration time, is set by the expected timescales for variation of the correlation result (~seconds) • Jij is an operator characterizing the net effect of the observing process for baseline (i,j), which we must calibrate • Sometimes Jij corrupts the measurement irrevocably, resulting in data that must be edited or “flagged”

21. Practical Calibration Considerations • A priori “calibrations” (provided by the observatory) • Antenna positions, earth orientation and rate • Clocks • Antenna pointing, gain, voltage pattern • Calibrator coordinates, flux densities, polarization properties • System Temperature, Tsys, nominal sensitivity • Absolute engineering calibration? • Very difficult, requires heroic efforts by observatory scientific and engineering staff • Concentrate instead on ensuring instrumental stability on adequate timescales • Cross-calibration a better choice • Observe nearby point sources against which calibration (Jij) can be solved, and transfer solutions to target observations • Choose appropriate calibrators; usually strong point sources because we can easily predict their visibilities • Choose appropriate timescales for calibration

22. “Absolute” Astronomical Calibrations • Flux Density Calibration • Radio astronomy flux density scale set according to several “constant” radio sources • Use resolved models where appropriate • Astrometry • Most calibrators come from astrometric catalogs; directional accuracy of target images tied to that of the calibrators (ICRF = International Celestial Reference Frame) • Beware of resolved and evolving structures and phase transfer biases due to troposphere (especially for VLBI) • Linear Polarization Position Angle • Usual flux density calibrators also have significant stable linear polarization position angle for registration • Relative calibration solutions (and dynamic range) insensitive to errors in these “scaling” parameters

23. A Single Baseline – 3C 286 Vis. Phase vs freq. (single channel) 120° 105° 3C 286 is one of the strong, stable sources which can be used as a flux density calibrator

24. Single Baseline, Single Integration Visibility Spectra (4 correlations) Vis. amp. vs freq. Vis. phase vs freq. Baseline ea17-ea21 Single integration – typically 1 to 10 seconds

25. Single Baseline, Single ScanVisibility Spectra (4 correlations) Vis. amp. vs freq. Vis. phase vs freq. baseline ea17-ea21 Single scan – typically 1 to 30 minutes, 5 to 500 integrations

26. Single Baseline, Single Scan (time-averaged)Visibility Spectra (4 correlations) Vis. amp. vs freq. Vis. phase vs freq. baseline ea17-ea21 Single scan – time averaged 2626

27. Baseline-based Cross-Calibration • Simplest, most-obvious calibration approach: measure complex response of each baseline on a standard source, and scale science target visibilities accordingly • “Baseline-based” Calibration • Calibration precision same as calibrator visibility sensitivity (on timescale of calibration solution). • Calibration accuracy very sensitive to departures of calibrator from known structure • Un-modeled calibrator structure transferred (in inverse) to science target!

28. Antenna-Based Cross Calibration • Measured visibilities are formed from a product of antenna-based signals. Can we take advantage of this fact? • The net signal delivered by antenna i, xi(t), is a combination of the desired signal, si(t,l,m), corrupted by a factor Ji(t,l,m) and integrated over the sky, and diluted by noise, ni(t): • Ji(t,l,m) is the product of a series of effects encountered by the incoming signal • Ji(t,l,m) is an antenna-based complex number • Usually, |ni |>> |si| - Noise dominated

29. Antenna-base Calibration Rationale • Signals affected by a number of processes • Due mostly to the atmosphere and to the the antenna and the electronics • The majority of factors depend onantenna only, not on baseline • Some factors known a priori, but most of them must be estimated from the data • Factors take the form of complex numbers, which may depend on time and frequency Atmospheric delay 2 Atmospheric delay 1 Instru-mental delay 1 Instru-mental delay 2 V output

30. Correlation of Realistic Signals - I • The correlation of two realistic signals from different antennas: • Noise signal doesn’t correlate—even if |ni|>> |si|, the correlation process isolates desired signals: • In the integral, only si(t,l,m), from the same directions correlate (i.e., when l=l’, m=m’), so order of integration and signal product can be reversed:

31. Correlation of Realistic Signals - II • The si& sjdiffer only by the relative arrival phase of signals from different parts of the sky, yielding the Fourier phase term (to a good approximation): • On the timescale of the averaging, the only meaningful average is of the squared signal itself (direction-dependent), which is just the image of the source: • If all J=1, we of course recover the ideal expression:

32. Aside: Auto-correlations and Single Dishes • The auto-correlation of a signal from a single antenna: • This is an integrated power measurement plus noise • Desired signal not isolated from noise • Noise usually dominates • Single dish radio astronomy calibration strategies dominated by switching schemes to isolate desired signal from the noise

33. The Scalar Measurement Equation • First, isolate non-direction-dependent effects, and factor them from the integral: • Here we have included in Jsky only the part of J which varies with position on the sky. Over small fields of view, J does not vary appreciably, so we can take Jsky = 1, and then we have a relationship between ideal and observed Visibilities: • Standard calibration of most existing arrays reduces to solving this last equation for the Ji

34. Solving for the Ji • We can write: • …and define chi-squared: • …and minimize chi-squared w.r.t. each Ji, yielding (iteration): • …which we recognize as a weighted average of Ji, itself:

35. Solving for Ji (cont) • For a uniform array (same sensitivity on all baselines, ~same calibration magnitude on all antennas), it can be shown that the error in the calibration solution is: • SNR improves with calibrator strength and square-root of Nant (c.f. baseline-based calibration). • Other properties of the antenna-based solution: • Minimal degrees of freedom (Nant factors, Nant(Nant-1)/2 measurements) • Constraints arise from both antenna-basedness and consistency with a variety of (baseline-based) visibility measurements in which each antenna participates • Net calibration for a baseline involves a phase difference, so absolute directional information is lost • Closure…

36. Antenna-based Calibration and Closure • Success of synthesis telescopes relies on antenna-based calibration • Fundamentally, any information that can be factored into antenna-based terms, could be antenna-based effects, and not source visibility • For Nant > 3, source visibility cannot be entirely obliterated by any antenna-based calibration • Observables independent of antenna-based calibration: • Closure phase (3 baselines): • Closure amplitude (4 baselines): • Baseline-based calibration formally violates closure!

37. Simple Scalar Calibration Example • Sources: • Science Target: 3C129 • Near-target calibrator: 0420+417 (5.5 deg from target; unknown flux density, assumed 1 Jy) • Flux Density calibrators: 0134+329 (3C48: 5.74 Jy), 0518+165 (3C138: 3.86 Jy), both resolved (use standard model images) • Signals: • RR correlation only (total intensity only) • 4585.1 MHz, 50 MHz bandwidth (single channel) • (scalar version of a continuum polarimetry observation) • Array: • VLA B-configuration (July 1994)

38. The Calibration Process • Solve for antenna-based gain factors for each scan on flux calibrator Ji(fd) (where Vij is known): Solve also gain factors for phase calibrator(s), Ji(nt) • Bootstrap flux density scale by enforcing constant mean power response: • Correct data (interpolate J as needed): true

39. Antenna-Based Calibration Visibility phase on a several baselines to a common antenna (ea17) 4141

40. J1822-0938 (calibrator) 3C391 (science) Calibration Effect on Imaging

41. How Good is My Calibration? • Are solutions continuous? • Noise-like solutions are probably noise! (Beware: calibration of pure noise generates a spurious point source) • Discontinuities indicate instrumental glitches • Any additional editing required? • Are calibrator data fully described by antenna-based effects? • Phase and amplitude closure errors are the baseline-based residuals • Are calibrators sufficiently point-like? If not, self-calibrate: model calibrator visibilities (by imaging, deconvolving and transforming) and re-solve for calibration; iterate to isolate source structure from calibration components • Any evidence of unsampled variation? Is interpolation of solutions appropriate? • Reduce calibration timescale, if SNR permits

42. A priori Models Required for Calibrators Point source, but flux density not stable Stable flux density, but not point sources

43. Antenna-based Calibration Image Result

44. Evaluating Calibration Performance • Are solutions continuous? • Noise-like solutions are just that—noise • Discontinuities indicate instrumental glitches • Any additional editing required? • Are calibrator data fully described by antenna-based effects? • Phase and amplitude closure errors are the baseline-based residuals • Are calibrators sufficiently point-like? If not, self-calibrate: model calibrator visibilities (by imaging, deconvolving and transforming) and re-solve for calibration; iterate to isolate source structure from calibration components • Mark Claussen’s lecture: “Advanced Calibration” (Wednesday) • Any evidence of unsampled variation? Is interpolation of solutions appropriate? • Reduce calibration timescale, if SNR permits • Ed Fomalont’s lecture: “Error Recognition” (Wednesday)

45. Summary of Scalar Example • Dominant calibration effects are antenna-based • Minimizes degrees of freedom • More precise • Preserves closure • Permits higher dynamic range safely! • Point-like calibrators effective • Flux density bootstrapping

46. Full-Polarization Formalism (Matrices!) • Need dual-polarization basis (p,q) to fully sample the incoming EM wave front, where p,q = R,L (circular basis) or p,q = X,Y (linear basis): • Devices can be built to sample these linear or circular basis states in the signal domain (Stokes Vector is defined in “power” domain) • Some components of Ji involve mixing of basis states, so dual-polarization matrix description desirable or even required for proper calibration

47. Full-Polarization Formalism: Signal Domain • Substitute: • The Jones matrix thus corrupts the vector wavefront signal as follows:

48. Full-Polarization Formalism: Correlation - I • Four correlations are possible from two polarizations. The outer product (a ‘bookkeeping’ product) represents correlation in the matrix formalism: • A very useful property of outer products:

49. The Matrix Measurement Equation • We can now write down the Measurement Equation in matrix notation: • …and consider how the Ji are products of many effects.

50. A Dictionary of Calibration Components • Ji contains many components: • F= ionospheric effects • T = tropospheric effects • P = parallactic angle • X = linear polarization position angle • E = antenna voltage pattern • D = polarization leakage • G = electronic gain • B = bandpass response • K = geometric compensation • Order of terms follows signal path (right to left) • Each term has matrix form of Ji with terms embodying its particular algebra (on- vs. off-diagonal terms, etc.) • Direction-dependent terms must stay inside FT integral • Full calibration is traditionally a bootstrapping process wherein relevant terms are considered in decreasing order of dominance, relying on approximate orthogonality