Spectrometer Pipeline

1. SPIRE Consortium MeetingLa Palma, Oct. 1 – 2 2008SPIRE FTS PipelineTrevor FultonBlue Sky Spectroscopy, Lethbridge, Canada

2. Spectrometer Pipeline AOTs • Point source/sparse map (SOF1/3) • A spectrum of a point source that is well centred on the central detectors of the FTS arrays and/or simultaneously obtain a sparse map of an area roughly 2'' in diameter. For sparse mapping of larger areas, a raster of point source observations will be made. • Field mapping (SOF2/4) • To take a spectrum of a region of sky or an extended source that is within the FOV of the spectrometer – i.e. less than 2.6'' circular. This is achieved by using the beam steering mirror to perform a low-frequency jiggle and observing multiple interferograms at each point of the jiggle pattern. For fully-sampled mapping of larger areas, a raster of multiple jiggle maps will be observed.

3. Pointing Building Block Observation Building Block Generic FTS Observation ... Time • At each pointing, an FTS observation is performed by scanning the spectrometer mechanism • The Spectrometer detector arrays are pointed at a target via a movement of the Herschel telescope (Raster) and/or the Beam Steering Mirror (Jiggle).

4. Spatial Sampling Options Sparse Intermediate Full NBSM=1 NBSM=4 NBSM=16

5. Spectral Sampling Options

6. Spectrometer Pipeline Overview • The flow of the SPIRE Spectrometer data processing pipelines has been designed to follow the spectrometer AOTs. • The overall structure of each pipeline maximizes the benefit of the redundant information that exists within each observation building block. • The data products that will be made available to observers are derived from the output of specific pipeline modules that are at the logical breaks in the overall pipelines.

8. FTS Pipeline Building Block Pipeline Modify Timelines Create Interferograms Modify Interferograms Transform Interferograms Modify Spectra

9. Signal Timeline Product • Level-0.5 Product that is the input to the FTS pipeline • This product contains a timeline of the recorded signal for each detector (Voltage vs. Time).

10. Modify Timelines Vd-RMS(t) Glitch template and threshold Remove Electrical Crosstalk Electrical crosstalk matrix 1st-Level Deglitching V4(t) V1(t) Reference Voltage Non-Linearity Correction Clipping Correction Conversion Factors V2(t) V5(t) LPF Components Time Domain Phase Correction Thermal Fluctuation Timeline Vth(t) Temperature Drift Correction Bolometer Time Constants V6(t) V3(t)

11. First Level Deglitching • Glitches are identified in the Detector Timelines by way of wavelet analysis (same as for Photometer 1st-Level Deglitching) • Signal samples that are flagged as glitches are corrected within the wavelet analysis. Vd-RMS(t) 1st-Level Deglitching Glitch template and threshold V1(t)

12. V1(t) Clipping Correction V2(t) Clipping Correction • Clipped signals are those whose raw ADC value = 0 or 216-1 (65535). • These signal samples are corrected by way of an 8th order polynomial fit to the neighbouring signal samples.

13. LPF Components V2(t) Time Domain Phase Correction Bolometer Time Constants V3(t) Time Domain Phase Correction • The combined effect of the low pass filters in the readout electronics and the thermal response of the bolometers gives rise to a delay in the measured signal samples. • The delay is quantified by a combination of the known filter parameters and empirically derived thermal time constants for each bolometer. • The delay per detector is removed by convolution.

14. Time Domain Phase Correction

15. V3(t) Remove Electrical Crosstalk Electrical crosstalk matrix V4(t) Remove Electrical Crosstalk • Electrical crosstalk assumptions: • Linear • Effects on primary detector are negligible • No x-talk between arrays

16. Non-Linearity Correction Reference Voltage V4(t) Non-Linearity Correction Conversion Factors V5(t) • Similar correction as that in the Photometer pipelines. • No Flux Conversion at this point.

17. Temperature Drift Correction • Vth(t), is derived from the thermometers (2 per BDA). V5(t) Temperature Drift Correction Correlation Parameters V6(t)

18. FTS Pipeline Building Block Pipeline Modify Timelines Create Interferograms Interferogram Creation Modify Interferograms Transform Interferograms Modify Spectra

19. Create Interferograms Position of ZPD z(t), P(t) V6(t) Interferogram Creation Obliquity Factor V6(x) • The mechanism timeline is divided into a set of individual timelines – one per spectrometer scan for the building block. • The mechanism timelines are then regularized by way of interpolation and converted from MPD to OPD.

20. Create Interferograms Position of ZPD z(t), P(t) V6(t) Interferogram Creation Obliquity Factor V7(x) • The input signal timelines for each detector are then merged with the regularized mechanism timelines to create a set of interferograms (one interferogram per scan per detector).

21. Create Interferograms V

22. Each Interferogram Product contains one interferogram (Voltage vs. Optical Path Difference) per scan per detector. Interferogram Product

23. Modify Interferograms V6(x) Reference Interferograms Telescope/SCAL Correction V7(x) Baseline Removal V8(x) Glitch Threshold Second Level Deglitching V9(x) Optical Phase Phase Correction V10(x) Apodization V11(x)

24. Telescope/SCAL Removal • The contributions to the derived interferograms from the telescope and from SCAL are removed from the measured interferograms by way of subtraction. V6(x) Telescope/SCAL Correction Reference Interferograms V7(x)

25. V7(x) Baseline Removal V8(x) Baseline Removal • The position-dependent baseline for each interferogram is first characterized • Low order polynomial • Low frequency components of the FT • The derived baseline is then removed from each interferogram by way of subtraction.

26. Baseline Removal

27. Second Level Deglitching • The interferograms for each detector are inspected and the statistical outliers are flagged as glitches. • Outliers are flagged on a positional basis using the MAD method. • Samples that have been deemed to be glitches are replaced by the average of the clean samples at that position for that detector. V8(x) Second Level Deglitching Glitch threshold V9(x)

28. Phase Correction • The symmetric portion of each interferogram is first transformed. • A low-order weighted fit is made to the measured in-band for each spectrum to quantify any phase that remains. • The phase is removed from the MR/LR spectra by multiplication and from the HR interferograms by convolution. V9(x) Phase Correction Optical Phase V10(x)

29. Phase Correction • The phase is removed from the MR/LR spectra by multiplication • The phase is removed from the HR interferograms by convolution. V9(x) Phase Correction Optical Phase V10(x)

30. V10(x) Apodization V11(x) Apodization • An apodization function is applied to each interferogram. • This reduces the effects of the Sinc ILS in the spectral to be derived.

31. Apodization

32. FTS Pipeline Building Block Pipeline • Modify Timelines • Create Interferograms • Modify Interferograms • Transform Interferograms • Fourier Transform • Modify Spectra

33. Transform Interferograms • Each interferogram in the building block is transformed into a spectrum in this module. • The spectral sampling interval will be fixed – the value of which will depend on the requested resolution V11(x) Fourier Transform V12(σ)

34. Transform Interferograms V11(x) Fourier Transform V12(σ)

35. Each Spectrum Product contains one spectrum (Voltage vs. Wavenumber) per scan per detector. Spectrum Product

36. Modify Spectra V12(σ) SpectralRSRF Spectral Response V13(σ) Conversion Factors Flux Conversion I14(σ) Optical Crosstalk Removal Optical Crosstalk Matrix I15(σ) Spectral Averaging I16(σ)

37. Spectral Response • The wavenumber-dependent RSRF for the Telescope→BDA path is removed from each of the measured spectra. V12(σ) Spectral Response Spectral RSRF V13(σ)

38. Flux Calibration • Compare the derived spectra with those from a source with a known flux. • The ratio between the two gives the wavenumber-dependant factor by which the measured spectra must be multiplied to give flux-calibrated quantities. V13(σ) Flux Calibration Flux Conversion Factors I14(σ)

39. Remove Optical Crosstalk • Now that the signal have been converted to units of optical power, an optical crosstalk correction matrix may be applied. I14(σ) Remove Optical Crosstalk Optical crosstalk matrix I15(σ)

40. Spectral Averaging • On a detector-by-detector and wavenumber-by-wavenumber basis, the spectra derived for all scans in the building block are averaged. • Outliers, flagged by MAD clipping, are optionally removed from the average (default setting is to remove outliers). I15(σ) Spectral Averaging I15(σ)

41. One Average Spectrum Product per Observation Building Block Each product contains one spectrum (Flux vs. Wavenumber) per detector Average Spectrum Product

42. Create Spectral Cube • The average spectrum products from each building block are merged together to form a single spectral cube. • The cube is sampled on a regular grid in both spatial dimensions as well as in the spectral dimension. ASP1(σ) ASP2(σ) ASP3(σ) ASPn(σ) … Spatial Regridding Spectral Cube (x, y, σ)

43. Create Spectral Cube

44. Conclusions • As for Photometer pipelines, the hard work is in producing good calibration files. • Spectrometer Pipelines are currently undergoing phase one of the Scientific Validation. • This includes are review of the supporting documentation • End-to-end pipeline test #4 (November 2008). • Complete outstanding development/calibration products (Winter 2008/2009). • Demonstration of the pipeline in its current implementation Thursday afternoon.