360 likes | 512 Views
Diffuse Reflection Imaging: Earthshine and other Faint Signals. Sam Hasinoff MIT CSAIL, TTIC, Google[x]. Samuel W. Hasinoff, Anat Levin, Philip R. Goode, and William T. Freeman, Diffuse Reflectance Imaging with Astronomical Applications,
E N D
Diffuse Reflection Imaging: Earthshine and other Faint Signals • Sam Hasinoff • MIT CSAIL, TTIC, Google[x] Samuel W. Hasinoff, AnatLevin, Philip R. Goode, and William T. Freeman, Diffuse Reflectance Imaging with Astronomical Applications, Proc. 13thIEEE International Conference on Computer Vision, ICCV 2011 http://people.csail.mit.edu/hasinoff/diffuse/
Joint work with: • Bill Freeman • MIT CSAIL • AnatLevin • Weizmann Institute • Philip Goode • Big Bear Solar • Observatory Special thanks to: Bernhard Schölkopf, Frédo Durand, LiviaIllie, David Chen
Indirect Imaging with Reflective Objects • specular • reflectance • virtual camera • glossy • reflectance • bad camera? • diffuse • reflectance • really bad camera?
Resolution of a Diffuse Reflector 1 Diffuse surfaces blur together incident lighting from many directions 0 -180° 0° +180° Lambertian reflectance • How much detail can we resolve?
Single-Bounce Light Transport 1 0 -180° 0° +180° surface albedo incoming radiance visibility BRDF clipped cosine reflected radiance . • reflectance = (albedo lighting) BRDF • for convex objects, distant illumination *
Linear Light Transport • Usual matrix formulation observed pixels transfer matrix distant lighting noise transfer matrix encodes geometry, BRDF, surface albedo
Resolution of a Diffuse Reflector • about 9 pixels? • images of a convex Lambertian objects with distant lighting lie in a 9D subspace (<2% average error) • [Basri and Jacobs, 2001] • [Ramamoorthi and Hanrahan, 2001] … e.g. lighting-insensitive recognition with 9 basis images
Key: Occlusion Geometry Codes Illumination convex about 9 pixels? self-occlusion potentially much better
Key: Occlusion Geometry Codes Illumination • What 3D shape lets us best resolve the lighting? convex about 9 pixels? self-occlusion potentially much better
Sculpture Design for Reflectance Imaging • self-occlusion preserves high frequencies • extreme: isolate rays by building pinhole camerasusing the scene • arbitrary fidelity (in theory), limited only by diffraction and geometric calibration double-pinholes • natural complexity • coded reflectance
Bayesian Reconstruction Method Standard MAP estimation • Gaussian prior [Wiener 1949] • Sparse derivative prior [Levin et al. 2007]
Reconstructing a Changing Target For time-varying lighting, reconstruct stable part • marginalize out temporal variations temporal binning with Gaussian prior temporal variations as correlated noise
Approach #1: Rim Reflectance Imaging • Exploit visibility changes near occlusion boundary • works for compact lighting, e.g. object lit head on • multiple orientations = tomography • +single shot enough • - need high resolution • - need accurate geometry not previously used in astronomy or computer vision
Approach #2: Time Varying Imaging • Exploit variation over time in the occlusion geometry • classical astronomy (“light curves”) • +single pixel enough • - need natural variation • - need many shots • - target might change e.g. surface of Pluto from Charon’s 1985-1990transits [Young et al. 1999]
Earth from Space from Earth - Feasibility Study Earth from Apollo 17, 1972 (NASA)
Low-Resolution Expectations NASA’s “Blue Planet” reconstruction we’d be happy with
Moon as Earth Reflector (“Earthshine”) Leonardo Da Vinci’s Codex Leicester (ca. 1510)
(Exaggerated) Geometry of Earthshine Earth north pole sunlight observer Moon from observer Moon north pole
(Exaggerated) Geometry of Earthshine Earth north pole Earthshine sunlight observer Moonshine Moon from observer Moon north pole
Integrating over Earth from the Moon albedo of “unknown” Earth patch cos(incoming) Earth phase (≈ 180° - Moon phase) cos(outgoing) Earth integrand, from point on the Moon (simulation) Aug 8, 2010, 5am Feb 24, 2010, 1am
Earthshine as Linear Transfer Matrix y = T ∙ x + n noise observations of earthshine Earth-Moon transfer matrix Earth image
Recipe for Earthshine Imaging • feasible observing times: • moon >2° above horizon (air mass) • moon <85% full (moonshine) • sun >8° below horizon (twilight) • clear skies • track moon - multi-minute exposures • > -2.5 app. mag. (V) • block moonshine glare with occluder BBSO [Qiuet al. 2003]
Approach #1: Visibility at Moon Rim • From rim of Moon, different Earth regions visible • Special challenges: • need very high resolution, SNR • need detailed moon topography • BRDF at grazing angles? Earthrise, as seen from Apollo 8 (NASA)
Best Case Earth-Bound Moon Imaging 8m telescope ESO Very Large Telescope (Chile) 0.07” resolution 26,000 pixels across lunar disk by comparison, HST is 0.05” (i.e. near diffraction limit for it 2.4m mirror) http://optics.org/article/9654/
Sampling the Earth and Moon Surface moon simulation, 2475 samples (55 per radial line) ground truth earth model (75x30 pixels) relative contribution, Lambertian Earth
Simulated Reconstruction for Single Shot 80 dB 60 dB 40 dB sparse prior reconstruction std. optics (0.4”) adaptive optics (0.07”)
Approach #2: Scanning over Time • Over time, Earth region visible to Moon & Sun changes • Single pixel observations is fine • Special challenges: • 1D set of viewpoints per day • time varying cloud cover, snow, etc. 1am, Feb 24, 2010 5am, Feb 24, 2010 2am, Aug 14, 2010 region of Earth visible from Moon’s disk, i.e. contributing to earthshine
Simulated Reconstruction over Time grd. truth contrib. recon. 1 night Aug 12-13, 2010 107 obs. (@3 min) 1 month Aug, 2010 412 obs. (@10 min) 1 year Jan-Dec, 2010 963 obs. (@60 min)
Simulated Reconstruction over Time grd. truth contrib. recon. 1 night Aug 12-13, 2010 107 obs. (@3 min) 1 month Aug, 2010 412 obs. (@10 min) 1 year Jan-Dec, 2010 963 obs. (@60 min)
Time-Varying Clouds • 50-75% of Earth covered by clouds • large-scale systems last about 3 days earth with ISCCP clouds next day clouds mean clouds const. mean cloud reconstruction naïve time-varying reconstruction covariant reconstruction simulated reconstruction (1 year @3min, 60 dB)
ReconstructingMars from Light Curves • direct observation • outer planet, so less phase change (>85% disk lit) • much less active weather than earth • axial tilt of 25° (2D directions) • more Lambertian? Martian surface
Martian Reconstruction from Historical Spectra relative contribution ground truth + visibility recon. from simulation (scaled) recon. from real data (hVB), 47 dB 234 spectra (single-pixel images) observed at 2 sites, from 1963-1965 [Irivine et al. 1968ab]
Simulation: Limits of Single-Pixel Mars Reconstruction 200 samples 400 samples 800 samples 1600 samples 3200 samples 23731 samples 1 Martian year (= 2 years) single-pixel images, 60 dB, no temporal variation 0.3% noise 0.1% noise 0.03% noise
10 Years of Moon Photos moonshine ∙ filter • 10 years • 40k+ photos • photos every 5 min • Big Bear Solar Observatory • (BBSO), near LA • single CCD (grayscale) Ib Ia earthshine Moon from BBSO [Qiuet al. 2003]
The Earthshine-Moonshine Ratio irradiance from “unknown” Earth moonshine ∙ filter < 1° Ib lunar phase 0° = full, 180° = new Ia atmosphere earthshine Earth moonshine reference: factors out viewing effects
Coda: Optimizing Capture Strategies • imaging tradeoffs, e.g. blur vs. noise • sensor characteristics vs. 2.8 dB 14.6 dB http://people.csail.mit.edu/hasinoff/