1 / 47

Emilio Kropff Thesis presentation September 19, 2007

Statistical and dynamical properties of large cortical network models: insights into semantic memory and language. Emilio Kropff Thesis presentation September 19, 2007. Potts Networks – Latching – Correlated patterns. Thesis presentation. September 19 th , 2007.

imani-warner
Download Presentation

Emilio Kropff Thesis presentation September 19, 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical and dynamical properties of large cortical network models: insights into semantic memory and language Emilio Kropff Thesis presentation September 19, 2007

  2. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Cerebral cortex – Braitenberg & Schüz, 1991 # of neurons >> # of input fibers Modifiable synapses No prefered direction in the connections Two-level associative memory with formation of cell assemblies Mostly excitatory synapses Great convergence & divergence Connections are very weak

  3. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Auto-associative memories - Pattern #2 active - Pattern #1 active - Pattern #3 active • No activity Hebbian Learning !!

  4. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Testing the memory - Pattern #2 active

  5. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Testing the memory Network damage

  6. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Two-level associative memory with formation of cell assemblies • Anatomical studies – Braitenberg & Schüz • Embodied theories of semantic memory • Feature representation

  7. Potts Networks – Latching – Correlated patterns state 1 state 2 state 0 … state 3 state S Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA The model: the trick • S: number of alternative states of a unit • S: number of alternative features of a unit • a (global sparseness): average number of units that are active in a global memory • a (global sparseness): average number of features describing a concept

  8. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Testing Kanter’s Potts network (Kanter,1988) • The patterns ξ are constructed randomly and stored in the network by modifying J. • The state of the network is set to some initial value (e.g: random or some stored memory if we want to test its stability). • A unit i is picked randomly and the fields hik are calculated. • The S states of unit i are updated following:

  9. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Reviewing and extending the results of Kanter, 1988 • Kanter’s result for low S is • We find high S behaviour is

  10. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Adding a zero state and sparseness a

  11. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Highly diluted approximation • Two units speak to each other with probability cM/N • Two states speak to each other with probability e

  12. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA

  13. The Kanter result for Potts networks has a logarithmic correction for high S. • If well defined, a sparse Potts network reaches an optimal storage capacity • In highly diluted networks this result applies, with  = p/(cM e) . • These results are in line with the conjecture of a limit in the amount of information per synapse that a network can store.

  14. Potts Networks – Latching – Correlated patterns + correlation latching! Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Dynamics of the network retrieval + adaptation time

  15. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA

  16. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA • In addition, the transition matrix is notsymmetric

  17. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Dynamics: latching of 2 patterns time

  18. Potts Networks – Latching – Correlated patterns • Units active in both patterns in different states • Units active in one of the two patterns • Unitsactive in both in the same state • Weakly: units active in both patternsin different states • Units active in one of the two patterns • Unitactive in both in the same state Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Dynamics: latching of 2 patterns ‘alternative features’ ‘pathological case’ c: shared units ‘shared features’ d: shared features

  19. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Does latching present a natural rudimentary grammar?

  20. Correlation seems to be at least one of the main properties determining latching. • However, the transition matrix is not symmetric, which means that there are other important factors. • The equilibrium between global inhibition and local self excitation can control the complexity of symbolic chains. • Three types of latching transition exist, each in a restricted region of parameters. Do they organize in time?

  21. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Category specific deficits • Patients were found with a significant impairment in their knowledge about living things (animals + foodstuffs) as opposed to manmade artifacts (Warrington & Shallice, 1984). • Impairment for nonliving has also been reported → double dissociation. Current ratio: 23% vs 77% (Capitani, 03) Theoretical accounts • The selective impairments respond to differences in the networks representing different categories: sensory/functional theory (Warrington & Shallice, 84), domain-specific hypothesis (Caramazza & Shelton, 98). • The network sustaining semantic memory is quite homogeneous but different categories have different typical correlation properties (McRee et al, 97; Tyler & Moss, 01; Sartori & Lombardi, 04).

  22. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Hopfield memories • If patterns are randomly correlated (Tsodyks,88), • However, if patterns have a non-trivial structure of correlations, the storage capacity colapses. • Solution #1:Orthogonalize the patterns before feeding the network. (i.e: Dentate Gyrus in Hippocampus) • In semantic memory correlation between stored patterns seems to play a major role.

  23. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Solution #2 ?? • We assume that pattern 1 is being retrieved • We split hi into the contribution of pattern 1 (signal) and the rest (noise) • We minimize the noise

  24. Classical result: hebbian learning supportsuncorrelated memories Jij= Σμ (ξiμ – a).(ξjμ – a) Jij= Σμ (ξiμ - ai).(ξjμ - aj) Classical result: catastrophe associated to correlated memories popularity: ak= 1/p Σμ ξkμ New result:a modification that supportscorrelated memories New result:the performance is the same with uncorrelated memories

  25. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with finite α, C ≈ ln(N) GAUSSIAN noise (If there is independence between neurons i and j).

  26. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with finite α, C ≈ ln(N) GAUSSIAN noise (If there is independence between neurons i and j).

  27. Potts Networks – Latching – Correlated patterns ... Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with finite α, C ≈ ln(N) F(x) (uncorrelated patterns) If F(x) decays fast enough

  28. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with finite α, C ≈ ln(N) F(x) If F(x) decays exponentially If F(x) decays fast enough If F(x) decays as a power law

  29. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Storing memories Damaging the network

  30. Potts Networks – Latching – Correlated patterns Connectivity 1 70 60 Performance 50 Number of neurons 40 30 Cc1 Cc2 20 0 C(# of afferent connections per neuron) 10 0.2 0.4 Popularity ai Entropy Sf= Σi ai (1-ai) summed over active neurons in the pattern Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Storage capacity ~ fixed p Cc

  31. Potts Networks – Latching – Correlated patterns non living living Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Category specific effects • McRae feature norms • 541 concepts described in terms of 2526 features • i=1 if feature i is included in the description of concept  and i =0 otherwise Probability Distribution Entropy Sf of objects

  32. When memories are correlated, they have variable degrees of ressistance to damage. • The robustness of a memory is inverse to how informative it is (Sf). • In addition, popular neurons affect negatively the general performance (decay of F(x)). • These results show how the current trend in category specific deficits (‘living’ weaker than ‘non living’) could emerge even in a purely homogeneous network.

  33. A singel cortical network with Potts units including addaptation and storing correlated patterns of activity in its long range synapses, presents all the properties studied in this thesis. Thank you!

  34. # of features popularity McRae’s feature norms • In the semantic memory literature, auto-associative networks are often presented as weak models. Why? • To convince psychologists one must show an auto-associative memory that is able to store feature norms.

  35. McRae’s feature norms Performance of the network theoretical prediction simulations Size of the subgroup of patterns

  36. a – average sparseness If – average information Number of patterns p in the subgroup McRae’s feature norms

  37. McRae’s feature norms Performance of the network theoretical prediction simulations Size of the subgroup of patterns

  38. McRae’s feature norms • Why the real network performs poorly? • Independence between features is not valid (e.g: beak and wings). Is this effect strong enough? In case it is, there would be a storage capacity colapse. • The system works but the approximation of diluted connectivity is not good.

  39. McRae’s feature norms: the full solution  + 2+ 3+ ...

  40. McRae’s feature norms: the full solution Performance of the network highly diluted full solution simulations Size of the subgroup of patterns

  41. 100 80 60 40 100 20 80 2490 2500 2510 2520 60 40 20 0 3000 4000 5000 6000 % of patterns retrieved a b Total number of neurons McRae’s feature norms: strategies to store more patterns 1-add unpopular neurons 2-eliminatepopular neurons

  42. McRae’s feature norms: strategies to store more patterns 4-popularity deppendent connectivity 3-recombination neurons i and j have high popularity: their coincidence will be less popular. If applied massively, this principle could change the whole distribution.

  43. Coding Consolidation The (episodic) memory pyramid popularity calculation final storage orthogonalization Hippo-campus Entorhinal cortex Perirhinal and parahippocampal cortex Unimodal and polymodal association areas Primary cortex: sensory motor areas

  44. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with α ≈ 0, C ≈ ln(N)

  45. Potts Networks – Latching – Correlated patterns ac Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with α ≈ 0, C ≈ ln(N) ac

  46. Potts Networks – Latching – Correlated patterns Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Propeties with α ≈ 0, C ≈ ln(N) • If you want to be an attractor, you should pick at least some unpopular units. • Lowering U can make any pattern retrievable -> ATTENTION

  47. Potts Networks – Latching – Correlated patterns • Units active in both patterns in different states • Units active in one of the two patterns • Unitsactive in both in the same state • Weakly: units active in both patternsin different states • Units active in one of the two patterns • Unitactive in both in the same state Thesis presentation. September 19th, 2007 Emilio Kropff, LIMBO-CNS-SISSA Dynamics: latching of 2 patterns ‘alternative features’ ‘pathological case’ c: alternative features ‘shared features’ d: shared features

More Related