Committee Update Building a visual hierarchy

1. Andrew Smith 30 July 2008 Committee UpdateBuilding a visual hierarchy

2. Outline • Confabulation theory • Summary • Comparisons to other AI techniques • Human Visual System • Building A Visual Hierarchy • Learning • Inference • Texture modeling (applications) • Future work (dissertation defence, Spring 2009)

3. Confabulation Theory • A theory of the mechanism of thought • Cortex/thalamus is divided into thousands of modules (1,000,000s of neurons). • Each module contains a lexicon of symbols. • Symbols are sparse populations (100s) of neurons within a module. • Symbols are stable states of a cortex-thalamus attractor circuit.

4. Confabulation theory (1/4) Key concept 1: Modules contain symbols, the atoms of our mental universe. Smell module: Apple, flower, rotten, … Word module: ‘rose’ ‘the’ ‘and’ ‘it’ ‘France’ ‘Joe’ … Abstract planning modules, etc. Modules are small patches of thalamocortical neurons. Each symbol is a sparse popuation of those neurons.

5. Confabulation theory (1/4)

6. Confabulation theory (2/4) Key concept 2: All cognitive knowledge is knowledge links between these symbols. Smell module: Apple, flower, rotten, … Word module: ‘the’ ‘and’ ‘it’ ‘France’ ‘Joe’ ‘apple’ … Only symbols that are meaningfully co-occurring may become linked.

7. Confabulation theory (3/4)

8. Confabulation theory (3/4) Key concept 3: A confabulation operation is the universal computational mechanism. Given evidence a, b, c pick answer x such that: x = argmaxx’ p(a, b, c | x’) We say x has maximum cogency.

9. Confabulation theory (3/4) • Fundamental Theorem of Cognition:[1] p(abgd|e)4 = p(abgde)/p(ae) ∙p(abgde)/p(be) ∙p(abgde)/p(ge) ∙p(abgde)/p(de) ∙p(a|e)p(b|e)p(g|e)p(d|e) If the first four terms remain nearly constant w.r.t e, maximizing the fifth term maximizes cogency (the conditional joint).

10. Confabulation theory (3/4)

11. Confabulation theory (4/4) Key concept 4: Each confabulation operation launches a control signal to other modules. Control mechanism of inference – studied by others in the lab. (not here)

12. Similarities to other AI / ML • Bayesian networks – a special case • A “confabulation network” is similar to a Bayesian Net with: • Symbolic variables (discrete & finite & exclusive state) with equal priors. • Naïve-Bayes assumption for CP tables. • Can use similar learning algorithms (counting for CPs) • Hinton’s (unrestricted) Bolzman Machines – generalized: • Do not require complete connectivity • (many) more than two states. • Can use stochastic (Monte Carlo) ‘execution’

13. Outline • Confabulation theory • Summary • Comparisons to other AI techniques • Human Visual System • A Visual Hierarchy • Learning • Inference • Texture modeling • Future Work (i.e. my thesis)

14. Human Visual System • Retina – “pixels” • Lateral Geniculate Nucleus (LGN) “center-surround” representation • Primary(…) Visual cortex (V1 …) • Simple cells: • Hubel Weisel (1959) • Modeled by Dennis Gabor features[] • Complex cells • more complicated (end-stops, bars, ???) Take inspiration for our first and second-level features

16. Outline • Confabulation theory • Summary • Comparisons to other AI techniques • Human Visual System • Building A Visual Hierarchy • Learning • Inference • Texture modeling • Future Work (i.e. my thesis)

17. Confabulation & vision • Features (symbols) develop in a layer of the hierarchy as commonly seen inputs from their inputs. • Knowledge links are simple conditional probabilities: • p(a|e) where a and e are symbols in connected modules) • All knowledge can therefore be learned by simple co-occurrence counting. • p(a|e) = C(a,e) / C(e)

18. Building a vision hierarchy • Can no longer use SSE to evaluate model • Instead, make use of generative model: • Always be able to generate a plausible image.

19. Data set • 4,300 1.5 Mpix natural images (BW)

20. Vision Hierarch – level “0” • We know the first transformation from neuroscience research: simple cells approximate Gabor filters. • 5 scales, 16 orientations (odd + even)

21. Vision Hierarch – level “0” • Does the full convolution preserve information in images? (inverted by LS) • Very closely.

22. Vision Hierarchy – level 1 • We now have a simple-cell like representation. • How to create a symbolic representation? • Apply principle: Collect common sets of inputs from simple cells: similar to a Vector Quantizer. • Keep the 5-scales separate • (quantize 16-dimensions, not 80)

23. Vision Hierarchy – level 1 • To create actual symbols, we use a vector quantizer • Trade-offs (threshold of quantizer) : Number of symbols Preservation of information Probability accuracy • Solution Use angular distance metric (dot-product) • Keep only symbols that occurred in training set more than 200 times, to get accurate p(a|e). • After training, ~95% of samples should be within threshold of at least one symbol. • Pick a threshold so images can be plausibly generated.

24. Vision Hierarchy – level 1 Oops! Ignoring wavelet magnitude makes all “texture features” equally prominent.

25. Vision Hierarchy – level 1 • Solution, use binning (into 5 magnitudes), then apply vector quantizers).

26. Vision Hierarchy – level 1 • ~10,000 symbols are learned for each of the 5 scales. • Complex features develop.

27. Vision Hierarchy – level 1 • Now image is re-represented as 5 “planes” of symbols:

28. Outline • Confabulation theory • Summary • Comparisons to other AI techniques • Human Visual System • Building A Visual Hierarchy • Learning • Inference • Texture modeling • Future Work (i.e. my thesis)

29. Texture modeling - Learning • We can now represent an image as five superimposed grids of symbols. • Transform data set • Learn which symbols are typically next to which. • (knowledge links)

30. Knowledge links: • Learn which symbols may be next to which symbols (conditional probabilities) • Learn which symbols may be over/under which symbols. • Go out to ‘radius’ 5.

31. Texture modeling – Inference 1 • What if a portion of our image symbol representation is damaged? • Blind spot • CCD defect • brain lesion • We can use confabulation (generation) to infer a plausible replacement.

32. Texture modeling – Inference 1 • Fill in missing region by confabulating from lateral & different scale neighbors (rad 5).

33. Texture modeling

34. Texture modeling

35. Texture modeling

36. Texture modeling • Conclusions • This visual hierarchy does an excellent job at capturing an image up to a certain order of complexity. • Given this visual hierarchy and its learned knowledge links, missing regions could plausibly filled in. This could be a reasonable explanation for what animals do.

37. Texture modeling – Inference 2 • Super-resolution: • If we have a low resolution image, can we confabulate (generate) a high-resolution version? • “Space out” the symbols, and confabulate values for the new neighbors

38. Texture modeling

39. Texture modeling

40. Texture modeling • Super-resolution: conclusions • Having learned the statistics of natural images, the generative properties of this hierarchy can confabulate (generate) plausible high-resolution versions of its input.

41. Outline • Confabulation theory • Summary • Comparisons to other AI techniques • Human Visual System • Building A Visual Hierarchy • Learning • Inference • Texture modeling • Future Work (Dissertation)

42. The next level… Level 2 symbol hierarchy • Collect commonly recurring regions of level 1 symbols. • Symbols at Level 2 will fit together like puzzle pieces. Thank you!