Compressive Sampling (of Analog Signals)

1. Compressive Sampling(of Analog Signals) Moshe Mishali Yonina C. Eldar Technion – Israel Institute of Technology http://www.technion.ac.il/~moshikomoshiko@tx.technion.ac.il http://www.ee.technion.ac.il/people/YoninaEldar yonina@ee.technion.ac.il Advanced topics in sampling (Course 049029) Seminar talk – November 2008

2. Context - Sampling Continuous signal Analog world Digital world Sampling A2D Reconstruction D2A

3. Compression “Can we not just directly measure the part that will not end up being thrown away ?” Donoho Compressed 392 KB15% Compressed 148 KB6% Compressed 950 KB38% Original 2500 KB100%

4. Outline • Mathematical background • From discrete to analog • Uncertainty principles for analog signals • Discussion

5. References • M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors," IEEE Trans. on Signal Processing, vol. 56, no. 10, pp. 4692-4702, Oct. 2008. • M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals," CCIT Report #639, Sep. 2007, EE Dept., Technion. • Y. C. Eldar, "Compressed Sensing of Analog Signals", submitted to IEEE Trans. on Signal Processing, June 2008. • Y. C. Eldar and M. Mishali, "Robust Recovery of Signals From a Union of Subspaces", arXiv.org 0807.4581, submitted to IEEE Trans. Inform. Theory, July 2008. # • Y. C. Eldar, "Uncertainty Relations for Analog Signals", submitted to IEEE Trans. Inform. Theory, Sept. 2008.

6. Mathematical background • Basic ideas of compressed sensing • Single measurement model (SMV) • Multiple- and Infinite- measurement models (MMV, IMV) • The “Continuous to finite” block (CTF)

7. Compressed Sensing “Can we not just directly measure the part that will not end up being thrown away ?” Donoho “sensing … as a way of extracting information about an object from a small number of randomly selected observations” Candès et. al. AnalogAudioSignal Nyquist rateSampling CompressedSensing Compression(e.g. MP3) High-rate Low-rate

8. Concept Goal: Identify the bucket with fake coins. Weigh a coinfrom each bucket Compression Nyquist: Bucket # numbers 1 number Weigh a linear combinationof coins from all buckets Compressed Sensing: Bucket # 1 number

9. Mathematical Tools non-zero entries at least measurements Recovery: brute-force, convex optimization, greedy algorithms, and more…

10. CS theory – on 2 slides Compressed sensing (2003/4 and on) – Main results is uniquely determined by Donoho and Elad, 2003 Maximal cardinality of linearly independent column subsets Hard to compute !

11. CS theory – on 2 slides Compressed sensing (2003/4 and on) – Main results is uniquely determined by Donoho and Elad, 2003 with high probability is random Donoho, 2006 and Candès et. al., 2006 Convex and tractable Donoho, 2006 and Candès et. al., 2006 Greedy algorithms: OMP, FOCUSS, etc. NP-hard Tropp, Cotter et. al. Chen et. al. and many other

12. Sparsity models unknowns measurements MMV Joint sparsity SMV IMV = Infinite Measurement Vectors (countable or uncountable) with joint sparsity prior How can be found ? Infinite many variables Infinite many constraints Exploit prior  Reduce problem dimensions

13. Reduction Framework Find a frame for Solve MMV Theorem Mishali and Eldar (2008) Deterministicreduction IMV MMV Infinite structure allows CS for analog signals

14. From discrete to analog • Naïve extension • The basic ingredients of sampling theorem • Sparse multiband model • Rate requirements • Multicoset sampling and unique representation • Practical recovery with the CTF block • Sparse union of shift-invariant model • Design of sampling operator • Reconstruction algorithm

15. Naïve Extension to Analog Domain Standard CS Discrete Framework Analog Domain Sparsity prior what is a sparse analog signal ? Generalized sampling Continuoussignal Operator Infinite sequence Finite dimensional elements Stability Randomness  Infinitely many Random is stable w.h.p Need structure for efficient implementation Reconstruction Finite program, well-studied Undefined program over a continuous signal

16. Naïve Extension to Analog Domain Standard CS Discrete Framework Analog Domain • Questions: • What is the definition of analog sparsity ? • How to select a sampling operator ? • Can we introduce stucture in sampling and still preserve stability ? • How to solve infinite dimensional recovery problems ? Sparsity prior what is a sparse analog signal ? Generalized sampling Continuoussignal Operator Infinite sequence Finite dimensional elements Stability Randomness  Infinitely many Random is stable w.h.p Need structure for efficient implementation Reconstruction Finite program, well-studied Undefined program over a continuous signal

17. “Success has many fathers …” Whittaker 1915 Nyquist 1928 Kotelnikov 1933 Shannon 1949 A step backward Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than

18. A step backward Every bandlimited signal ( Hertz) can be perfectly reconstructed from uniform sampling if the sampling rate is greater than Fundamental ingredients of a sampling theorm • A signal model • A minimal rate requirement • Explicit sampling and reconstruction stages

19. Discrete Compressed Sensing Analog Compressive Sampling

20. no more than N bands, max width B, bandlimited to • More generally only sequences are non-zero Analog Compressed Sensing What is the definition of analog sparsity ? • A signal with a multiband structure in some basis • Each band has an uncountable number of non-zero elements • Band locations lie on an infinite grid • Band locations are unknown in advance (Mishali and Eldar 2007) (Eldar 2008)

21. Multi-Band Sensing: Goals bands Sampling Reconstruction Analog Infinite Analog Goal: Perfect reconstruction Constraints: • Minimal sampling rate • Fully blind system What is the minimal rate ? What is the sensing mechanism ? How to reconstruct from infinite sequences ?

22. Rate Requirement • The minimal rate is doubled. • For , the rate requirement is samples/sec (on average). Theorem (blind recovery) Mishali and Eldar (2007) Theorem (non-blind recovery) Landau (1967) Average sampling rate • Subspace scenarios: • Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) • Half blind system (Herley and Wong 99, Venkataramani and Bresler 00)

23. Sampling Multi-Coset: Periodic Non-uniform on the Nyquist grid In each block of samples, only are kept, as described by 2 Analog signal 0 Point-wise samples 0 3 3 2 0 3 2 Bresler et. al. (96,98,00,01)

24. The Sampler in vector form unknowns Length . known matrix known Observation: is sparse DTFT of sampling sequences Constant Problems: • Undetermined system – non unique solution • Continuous set of linear systems is jointly sparse and unique under appropriate parameter selection ( )

25. Paradigm Solve finiteproblem Reconstruct 0 S = non-zero rows 1 2 3 4 5 6

26. Continuous to Finite Solve finiteproblem Reconstruct CTF block MMV • span a finite space • Any basis preserves the sparsity Continuous Finite

27. Algorithm Perfect reconstruction at minimal rate Blind system: band locations are unkown Can be applied to CS of general analog signals Works with other sampling techniques Continuous-to-finite block: Compressed sensing for analog signals CTF

28. Blind reconstruction flow Multi-coset with Universal SBR4 Yes CTF No SBR2 No Bi-section CTF Yes Uniform at Ideal low-pass filter Spectrum-blind Sampling Spectrum-blind Reconstruction

29. Final reconstruction (non-blind) Bresler et. al. (96,00)

30. Framework: Analog Compressed Sensing Sampling signals from a union of shift-invariant spaces (SI) Subspace generators

31. Framework: Analog Compressed Sensing There is no prior knowledge on the exact indices in the sum What happen if only K<<N sequences are not zero ? Not a subspace ! Only k sequences are non-zero

32. Framework: Analog Compressed Sensing Step 1: Compress the sampling sequences Step 2: “Push” all operators to analog domain CTF System A High sampling rate = m/TPost-compression Only k sequences are non-zero

33. Framework: Analog Compressed Sensing Low sampling rate = p/TPre-compression System B CTF Theorem Eldar (2008)

34. Does it work ?

35. Minimal rate Minimal rate Simulations Sampling rate Sampling rate Brute-Force M-OMP

36. Simulations (2) 0% Recovery 100% Recovery 0% Recovery 100% Recovery Noise-free Sampling rate Sampling rate SBR4 SBR2 Empirical recovery rate

37. Simulations (3) Signal Reconstruction filter Amplitude Amplitude Output Time (nSecs) Time (nSecs)

38. Break (10 min. please)

39. Uncertainty principles • Coherence and the discrete uncertainty principle • Analog coherence and principles • Achieving the lower coherence bound • Uncertainty principles and sparse representations

40. The discrete uncertainty principle Uncertainty principle

41. Discrete coherence Which bases achieve the lowest coherence ?

42. Discrete coherence Which signal achieves the uncertainty bound ? Spikes Fourier

43. Discrete to analog • Shift invariant spaces • Sparse representations • Questions: • What is the analog uncertainty principle ? • Which bases has the lowest coherence ? • Which signal achieves the lower uncertainty bound ?

44. Theorem Eldar (2008) Analog uncertainty principle Theorem Eldar (2008)

45. Bases with minimal coherence In the DFT domain Fourier Spikes What are the analog counterparts ? • Constant magnitude • Modulation • “Single” component • Shifts

46. Bases with minimal coherence In the frequency domain

47. Tightness

48. Sparse representations • In discrete setting

49. Sparse representations • Analog counterparts Undefined program ! But, can be transformed into an IMV model

50. Discussion • IMV model as a fundamental tool for treating sparse analog signals • Should quantify the DSP complexity of the CTF block • Compare approach with the “analog” model • Building blocks of analog CS framework.