1 / 55

The perils of bidirection al excitation..

Explore the benefits and challenges of bidirectional excitation in neural networks, including pattern completion, imagery, and amplification. Discover the problems that can arise with bidirectional excitation and how inhibition plays a crucial role in regulating activation. Learn about the mechanisms and types of inhibition, including feedback and feed-forward inhibition, and alternative inhibition functions like k-Winners-Take-All (kWTA).

bettinac
Download Presentation

The perils of bidirection al excitation..

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The perils of bidirectionalexcitation.. • Advantages in terms of pattern completion, imagery, amplification, etc • But can you think of problems that can arise with bidir excitation? • If features turn categories on, and categories turn on features, and….

  2. Inhibition: motivating questions Q: Why not turn everything on all the time? A: That’s like having everything off, but wasteful! Q: What’s different about inhibitory and excitatory neurons? A: Lots, but mostly where they project (send info) to. Generally... • Excitatory neurons project locally & to different areas • Inhibitory neurons primarily project within small regions • Excitatory neurons carry information for communication • Inhibitory neurons are responsible for (locally) regulating the activation of excitatory neurons.

  3. Inhibition: ion mechanism Who remembers how this works? What were the primary NTs? Glutamate is excitatory: • Opens Na+ channels • Na+ enters cell, increasing Vm GABA is inhibitory: • Opens Cl- channels • Cl- enters cell, increasing Vm…? • ...only if Vm > Einhib. reversal

  4. Inhibition: Uses • Controls activity (bidirectionalexcitation). • Inhibition as athermostat-controlled airconditioner: • Inhibitoryneuronssampleexcitatoryactivity(likeathermostat samples thetemperature) • Moreexcitatoryactivity→moreinhibitiontokeepthenetworkfrom gettingtoo“hot”(active)→setpointbehavior

  5. Types ofInhibition Feedback b) a) Hidden Inhib Hidden Inhib Feed− Forward Input Input Anticipatesexcitation Reacts toexcitation

  6. Types ofInhibition Feedback b) a) Hidden Inhib Hidden Inhib Feed− Forward Input Input Anticipatesexcitation Reacts toexcitation Like havingthermostat outside of yourhouse Like a normal(indoor) ACthermostat

  7. <Inhib.proj> parameter explanation Feed-forward inhibition • Strength of FF weights to inhib (ff_wt_scale) • How much the input turns on inhibition Feed-back inhibition • Strength of FB weights to inhib (fb_wt_scale) • Hidden units turning on inhibition Both FF and FB • Inhib conductance into inhib units (g_bar_i.inhib) • Inhibitory units inhibiting each-other • Inhib conductance into hidden units (g_bar_i.hidden) • Inhibitory units inhibiting hidden units

  8. Simulations:[inhib.proj]

  9. FFFB Inhibition function • Can summarise effects of both types of inhibition by manipulating inhibitory conductance in all units within a layer in a way that would approximate what interneurons would do (while retaining dynamics). • This lets us avoid simulating inhibitory interneurons, which is a big computational saving • Also “cuts to the chase” and avoids oscillations in inhibitory response • In some cases inhibitory neurons may be of interest, and we can still simulate them where desirable. Just don’t need to in every case.

  10. FFFB inhibitionfunction • Wecanapproximatefeedforward(FF)andfeedback(FB)aspectsof inhibitoryinterneuronsusingtheFFFBinhibitionfunction: • averagenetinput:<η>=Ln1ηi n • averageactivation:<y>=Ln1yi n • Then:ff(t)=ff[<η>−ff0]+ • fb(t)=fb(t−1)+dt[fb<y>−fb(t−1)] • NowjustsetgiintargetlayerasafunctionofFFandFB: • gi(t)=gi[ff(t)+fb(t)] Advantages: Much less computationally expensive, avoidsoscillations

  11. Simulations: [inhibfffb.proj] FFFB approximates set pointbehavior. Allowsforfasterupdating,reducesoverallcomputation. Canuseinlargenetworkswithmultiplelayers,withinhibition summarized byFFFB Canstillcapturedifferentialamountsofinhibitionindifferentbrain areas with FFFB params: gi, FF andFB components insomeapplicationsmaystillwantactualinhibneurons

  12. Competition • Inhibition within a layer creates competition • Why might this be useful? • Forces different units to do different things • Turns off noisy (irrelevent) units for a given computation • Onlythemost appropriate(best-fitting)unitssurvivethecompetition

  13. Alternative inhibition function(optional): k-Winners-Take-All(kWTA)

  14. Alternative inhibition function(optional) k-Winners-Take-All(kWTA) • Thefunctionofinhibitionistokeepexcitatoryactivityataroughset point. • Wecanapproximatethisfunctionbyenforcingamaxactivitylevelin eachlayer. • kWTA: Instead of simulating inhibitory neurons, we choose an inhibitorycurrentgi valueforeachlayersuchthatthespecified numberkofexcitatoryneuronsareabovethreshold.

  15. Alternative inhibition function(optional) k-Winners-Take-All(kWTA) • Thefunctionofinhibitionistokeepexcitatoryactivityataroughset point. • Wecanapproximatethisfunctionbyenforcingamaxactivitylevelin eachlayer. • kWTA: Instead of simulating inhibitory neurons, we choose an inhibitorycurrentgi valueforeachlayersuchthatthespecified numberkofexcitatoryneuronsareabovethreshold. • Advantages: Much less computationally expensive, avoidsoscillations.

  16. kWTA:Summary • Simpleshortcutweuseinsteadofactualinhibitoryinterneurons • Capturesbasicideathatinhibitionmaintainsactivityata • set point for a givenlayer • Specifyinhibitionvalueforalayersuchthatkunitsareactive • kisaparameter:percentactivitylevelsvaryacrossdifferentbrain regions! • kWTAstillallowsforsomewiggleroominoverallactivation

  17. Benefits ofInhibition • Controls activity (bidirectionalexcitation) • Inhibitionforcesunitstocompetetorepresenttheinput:Onlythemost appropriate(best-fitting)unitssurvivethecompetition

  18. Networks • Biology: The cortex • Excitation: • Unidirectional(transformations) • Bidirectional (top-down processing, patterncompletion, amplification) • Inhibition: Controls bidirectional excitation (feedforward, feedback,set point, FFFBapproximation) • Constraint Satisfaction: Putting it alltogether.

  19. ConstraintSatisfaction

  20. ConstraintSatisfaction Processoftryingtosatisfyvariousconstraints(fromenvironment, connection weights,activations). Bidirectionalexcitationandinhibitionformpartofthislarger computationalgoal.

  21. ConstraintSatisfaction Processoftryingtosatisfyvariousconstraints(fromenvironment, connection weights,activations). Bidirectionalexcitationandinhibitionformpartofthislarger computationalgoal. Energy/harmony. AttractorDynamics. Noise.

  22. Harmony Harmony=extenttowhichunitactivationsareconsistentwithweights H=1LjLiaiwijaj 2

  23. Harmony Harmony=extenttowhichunitactivationsareconsistentwithweights H=1LjLiaiwijaj 2 Harmonyishighwhenunitswithstrong(positive)weightsareco-active

  24. Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles

  25. Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles

  26. Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles networksettling=movingtoamore“harmonious”state

  27. Attractors An attractor network is a network of neurons with excitatory interconnectionsthatcansettleintoastablepatternoffiringgivenarange of different startingstates.

  28. Attractors An attractor network is a network of neurons with excitatory interconnectionsthatcansettleintoastablepatternoffiringgivenarange of different startingstates. [hereweconsideronlyfixedpointattractors,butcyclicalorchaotic attractors are alsopossible]

  29. AttractorDynamics Bidirectionalexcitationcausesanetworktosettleintoaparticularstable state over time: theattractor. Cool Demos on Hopfield attractors: http://jackterwilliger.com/attractor-networks/

  30. AttractorDynamics Bidirectionalexcitationcausesanetworktosettleintoaparticularstable state over time: theattractor. Circle indicates attractor basin. Maximizeharmonygiveninputsandweights.

  31. The NeckerCube a) b) c) • Two differentinterpretations • Can’t perceive both atonce • Alternate between perceptions:bistability

  32. Otherexample

  33. The Role ofNoise Howmightnoisebeusefulinyourbrain?

More Related