CRUSH

1. CRUSH OPTIONS IMAGES

2. The Submillimeter Challenge @350 um Atmospheric Background is 107 times brighter than faint galaxies. Analogous to Observing a ~16 Magnitude star at daytime!

3. The Atmospheric Power Spectrum (5 January 2003) Strong 1/f characteristic Turnover to white noise regime at ~ 10 Hz Need Fast Sampling of Background (SHARC-2 has samples every 36ms)

4. Differencing of Signals 100 Jy source

5. Residual Sky Noise Chopped Lissajous Sweep Vesta ~5 Jy (2 min) Simulated 4 Hz chopper with 40” throw under better than average conditions Blank Sky (10 min) Chopped Image has Limiting noise typically 2-3 times higher than 'state of the art' reduction. Not including deconvolution noise!!!

6. The Need for Speed (Sweeping) • Moving Several pixels onto the same sky position within the limiting instrumental time scale 'calibrates' pixels against one another. • The more pixels that can be thus related, the more robust the 'calibration' measurement. • 'Calibrated' pixels can observe several positions on the sky within the limiting time scale, leading to high fidelity maps. Faster • Better pixel-to-pixel calibration • Larger and higher fidelity maps • Crossing Sweep Patterns • Non-periodic source crossing • Move Primary to avoid changing illumination pattern Smarter and Better

7. Scanning Strategies without a Chopping Secondary For large map making. Obtains uniform coverage over an area much larger than the array For compact and point sources Maximizes time coverage over a small area.

8. From Chopped Data to Discreet Modeling Singular Value Decomposition Mathematically Rigorous Maximum Entropy Solution. Difficulties with SVD • Computationally costly. (Large Matrices to invert) • Non-linearities. (Gain fitting). • Degeneracies, Singularities and Constraints • Time dependent noise Parallel SVD Effort at Goddard Space Flight Centre

9. Model 1 Iterative Reduction • Series of Maximum Likelihood Estimators. • Brightest First • Convergence via Iterating Model 2 Advantages Model 3 • Intuitive • Fast. (Linear with Data Size) • Able to Deal with Non-Linearities • Easily Configurable / Changeable Model N Final Source Model

10. Instrument Specific Models Vesta 8293 5 January 2003 Excellent Conditions Static Residual Pixel Offsets ~ 2000 Jy Static Residual Pixel Offsets ~ 2000 Jy Source Model ~ 5 Jy = Electronic Row Drifts ~ 1 Jy Detector 1/f Drift Model ~ 1 Jy Acceleration Response ~ 0.2 Jy

11. The Format of the SHARC-2 Array 32 x 12 pixels. Nearly Nyquist Sampled.

12. Maximum Likelihood Estimators Alternatively, in terms of the Residuals: As if residuals only contained given model...

13. CRUSH Models with Maximum Likelihood Estimators Typical Time Scale scan 10 min scan scan 1 chop* 1 frame 1 second ~30 seconds ~1 second 0.1-1 second scan all time scales all time scales Typical Spacial Scale pixel array pixel pixel pixel array row pixel array 4x4 to 8x6 pixel pixel pixel Typical Flux 1,000 – 10,000 Jy 10-1000 Jy ~ 10 Jy 1-10 Jy ~1 Jy ~1 Jy ~10 mJy ~10 mJy Model Name Residual Pixel Offsets Correlated Background Noise Gain Modelling Pixel Weights Chopping Residuals. Temperature Gradients. Electronic Row Drifts Detector 1/f Drifts Time Weights Residuals Spikes Regional Correlations Acceleration Response Temporal Features Spectral Features

14. Simulated Raw Data (1/f correlated) SHARC 1.5 Uranus Raw (Dowell) Preliminary Simulations for the Iterative Reduction Method (Feb 2001) Partial Cleaning (50 iterations) SHARC 1.5 Uranus Reduced (Dowell) Reduction Goal Simulated source with only white noise – this is the best any analysis could achieve Deep Cleaning (200 iterations) SHARC 1.5 – single row of bolometers. Edge pixels used for estimating correlated noise.

15. Simulations with CRUSH Tom Tyranowski 100 mJy Ring surrounding compact star in 1 hour 10' x 10' Billiard Ball Scan. Imperfect cleaning of faint large scale structures 100 mJy Compact Lissajous Sweep in 1 hour Clean background on small scales Source Fluxes Recovered within 1%

16. Correlated Noise And Gain Fitting

17. Non-Linear Response where, Small signal gain Large signal gain

18. Weighting Weights Separated into a product of pixel-only weights and time-only weights. Where B is the number of Active Bolometers, and P is the number of Parameters fitted in the time interval T' Where P is the number of fitted parameters in the time interval T

19. Identifying Anomalous Pixel Behaviour • One of the Most challenging aspects of deep reductions • Removal of Residual Spikes • Flagging of Pixels with Unreasonable Gain Fits • Looking for Statistically Significant Temporal Features... • ... and Spectral Features About 300 of 384 pixels are good enough

20. The Lockman Hole z ~ 2-3 SHARC-2 • Follow-up of SCUBA galaxies • 4 Fields, 3-4 Hours of Grade I Each • 1-sigma of ~ 5mJy depth. • 3 of 4 SCUBA galaxies detected • Additional 7 objects detected above noise

21. Map Noise Characteristics & Detection Confidence • Perfectly Gaussian Noise • ~2 Times wider than expected statistically from independent pixel noise! Detectors are NOT independent! • Positive tail Clearly indicating the presence of source flux.

22. Residual Pixel-to-Pixel Covariances (Learning from the Data Themself...) Gaussian with ca. 35” FWHM

23. Principal Options to CRUSH • Reduction Options • Scan Specific Options • Model Specific Options • Source Map • Gain Modelling • Frame and Pixel Weighting • Row Model • Detector Drift Model • 2-D Gradient Model • Residual Spike Removal • Temporal and Spectral Feature Identification • Chopping Residual Model • Acceleration Response Model

24. Options to CRUSH crush [options] -help A concise listing of ALL available options to crush, including their current settings. http://www.submm.caltech.edu/~sharc/crush/glossary.html A detailed explanation of the options, with corresponding configuration keys, and valid arguments. Cross-referenced and easily searchable. The true “CRUSH User's Guide”.

25. Principal Reduction Options Brightness Related -faint -deep LOAD_CONFIG = faint.cfg LOAD_CONFIG = deep.cfg < 1 Jy < 100 mJy. Similar to -faint -compact, but more aggressive. Size Related Do not worry about filtering extended structures on the scale of the array. Instead clean aggressively... EXTENDED_STRUCTURE = false -compact Miscalleneous -rounds= -point -config=xxx -altaz ITERATIONS LOAD_CONFIG =point.cfg LOAD_CONFIG =xxx ALTAZ =true The number of iterations before final solution. Perform pointing after reduction. Load setting for confguration file xxx. Reduce maps in Alt/Az coordinates instead of the default RA/DEC.

26. Scan Specific Options Options affect all scans that are subsequently listed. Preconditioning of Scan data -average= -chopped= AVERAGE_FRAMES CHOPPED_OBSERVATION = true The number of 36 millisecond frames to integrate data over. E.g. -average=3. If a chopping secondary was used. Adjustments Scale the scan data by this factor. E.g. -scale=1.23 13852 13853 Adjust the pointing. E.g. -FAZO=-103.0 -FZAO=12.7 9712 Change the tau value to use. E.g -tau=225GHz:0.046 15224 -scale= -FAZO= -FZAO= -tau=id:value Location Specify the directory where the raw data resides. E.g. -path=/home/nobody/data/SHARC2 RAW_DATA_PATH -path=

27. Generic Options Activation Iteration -Ixxx= XXX_TURN Specify when a given model is to be activated. Several models have 'auto' setting available. When specified each model will evaluate a number of possible criteria, based on which it activates or delays. One can also explicitly activate a model at the given iteration. Characteristic Time Constant -xxxT= XXX_T = Several of the models have time constants assigned to them. Most commonly they control the time resolution of the given model. The longer the time constant, the less often a parameter is fitted for the given model. Large ~ Robust Small ~ Aggressive

28. Options to Source Map Pipeline Auto Configure Related -Ifidel= -Iextended= MAP_FAITHFUL_TURN EXTENDED_STRUCTURE_TURN Do not allow self-activating models to activate until the given iteration. Also, if negative clipping is enabled, clip images until the specified iteration. E.g. -Ifidel=3 If EXTENDED_STRUCTURE is set true, do not self activate models that may remove extended structure until the specified number of source generations is obtained. E.g. -Iextended=4 Map Appearance Related Clip noisy map edges. Use exposure time relative to peak exposure to define clipping criterion. E.g. -minExp=0.1 Convolve map to beam size to get rid of unwanted high spacial frequency structure. E.g. -convolve=4.0 MAP_RELATIVE_EXPOSURE = CONVOLVING_BEAM_FWHM = -minExp= -convolve=

29. Gain Fitting and Gain Adjustment Options Gain Related -gainRounds= -gainGoal= GAIN_ITERATIONS GAIN_CONVERGENCE_GOAL The number of initial gain iterations. Like the full pipeline iterations, except only gain and models preceding gain are solved for. The desired gain convergence if GAIN_ITERATIONS is set to 'auto'. The fainter the source, the better the convergence that is required. Fast Line-of-Sight Opacity Adjustment Allow/Disallow for fast (real-time) adjustment of the extinction around the mean extinction value (explicitly defined or obtained from Mai-Tau) base on the background power variations. TAU_ADJUST=true/false -tauAdjust -notauAdjust

30. Weighting and Flagging Weight Related -pWeightT=min-max Set the time scales over which pixel weights are to be determined. Max should be set such that it is smaller than beam crossing time. PIXEL_WEIGHT_T Flagging Related -minDOF= -spikeLevel= -spikeRatio= -maxTemporal= -maxSpectral= The minimu degrees of freedom that must remain per pixel for the pixel to remain active. Remove residual spikes above the specified significance level. Flag pixels that have a higher fraction of spikes to data point. Flag pixels with temporal features above the spec'd significance level. Flag pixels with spectral features above the spec'd significance level. MIN_DEGREES_OF_FREEDOM DESPIKE_LEVEL REJECT_SPIKE_FRACTION MAX_TEMPORAL_FEATURE MAX_SPECTRAL_FEATURE

31. Output Related Options Gain Related -outpath= -name= -precess= -resolution= The output directory in which to place files that are automatically names as [SourceName].[Scan1#]. ... .[ScanN#].fits. E.g. -outpath=/home/nobody/reduced The name of the output file including absolute path. E.g. -name=/home/nobody/reduced/mysource.fits The coordinate epoch in which the reduced map is written. CRUSH does precession!!! E.g. -epoch=1950.0 The map resolution. The setting 'auto' defaults to 1/3 Cassegrain detector pixel size, ~1”.62, but one can set this to any desired value really. REDUCED_MAP_PATH REDUCED_MAP_EPOCH MAP_RESOLUTION

32. CRUSH Tips • Reduce as much data at once as you possibly can.Coadd later. Increase the memory allocated to the Java VM. • Reduce Data together that were taken with different Position Angles and Scanning Angles. • If default reduction is 'funny'. • Try different brightness. (-bright, -faint, -deep) • Adjust gain convergence criterion. • Try -compact if emission is not on the scale of the array. • Adjust specific model time-scales. • Try to identify problematic scans. Act on them! • Trade Extended flux for Flatter Baselines. • Check on Pointing. • Use Mai-Tau.

33. Observing Tips • Choose Appropriate Scanning Pattern • Size (smaller patterns yield cleaner data) • Coverage (Lissajous vs Billiard Ball) • Speed (Faster = Better) • Orientation (not along rows!!!) • Feasibility. • Chopping / Not Chopping... • Point Often on Nearby object (esp. in ZA) • Calibrate Often on known calibrator • Check on Beam Quality every night • Use DSOS (Dish Surface Optimization System) • Keep Detailed Logs

34. The Data Structure * Free FITS viewing software FV

35. Primary Image: Flux Distribution “Measurement Flux” As would be seen by detector if there was no atmospheric absorbtion Natural units are Voltage (nV, V) Response to 1 Jy source Pseudo flux units (Jy/beam, Jy/sr, Jy/arcsec**2)

36. Aperture Flux For some Aperture With default crush pixelization (1/3 Cassegrain Pixel Size)

37. Peak Flux – (Flux Inside a Beam) Where, the typical 350um values for SHARC-2 are,

38. Smoothing with a Beam Define Smoothing as Where, the a typical smoothing beam will be a Gaussian of the form:

39. Second Image: RMS “Measurement Uncertainty” Uncertainty of the Detector Measurement “Map Uncertainty” The Uncertainty of the Flux value on the map. For default pixelization... ...and Gaussian Beam

40. Flux Uncertainty inside an Aperture With default crush pixelization (1/3 Cassegrain Pixel Size)

41. Excess Noise and Non-Independent Pixels Correlated Pixel Noise Some Linear Quantity A The Uncertainty on A

42. Excess Noise Determination From Map Chi-Squared(No source) Gaussian Fit upper limit From Map NoiseDistribution

43. Crush Suite of Utilities imagetool Aiming to be a comprehensive image manipulation tool. Scaling, rms scaling, map units, regridding, smoothing, filtering and clipping. Under development... show (via imagetool) Display tool that allows quick view of images with some limited capabilities such as PSF fitting, image toggle, units on the fly etc. coadd Produce maps out of multiple images. jiggle Shift maps on the fly to determine alignment covarsee A visualization tool for pixel-to-pixel covariance matrixes. histogram Map signal-to-noise Histrogram generator. Useful for determining excess noise + quick diagnostic deconvolve Obtain super-resolution. Undocumented...