550 likes | 563 Views
Explore the benefits and challenges of bidirectional excitation in neural networks, including pattern completion, imagery, and amplification. Discover the problems that can arise with bidirectional excitation and how inhibition plays a crucial role in regulating activation. Learn about the mechanisms and types of inhibition, including feedback and feed-forward inhibition, and alternative inhibition functions like k-Winners-Take-All (kWTA).
E N D
The perils of bidirectionalexcitation.. • Advantages in terms of pattern completion, imagery, amplification, etc • But can you think of problems that can arise with bidir excitation? • If features turn categories on, and categories turn on features, and….
Inhibition: motivating questions Q: Why not turn everything on all the time? A: That’s like having everything off, but wasteful! Q: What’s different about inhibitory and excitatory neurons? A: Lots, but mostly where they project (send info) to. Generally... • Excitatory neurons project locally & to different areas • Inhibitory neurons primarily project within small regions • Excitatory neurons carry information for communication • Inhibitory neurons are responsible for (locally) regulating the activation of excitatory neurons.
Inhibition: ion mechanism Who remembers how this works? What were the primary NTs? Glutamate is excitatory: • Opens Na+ channels • Na+ enters cell, increasing Vm GABA is inhibitory: • Opens Cl- channels • Cl- enters cell, increasing Vm…? • ...only if Vm > Einhib. reversal
Inhibition: Uses • Controls activity (bidirectionalexcitation). • Inhibition as athermostat-controlled airconditioner: • Inhibitoryneuronssampleexcitatoryactivity(likeathermostat samples thetemperature) • Moreexcitatoryactivity→moreinhibitiontokeepthenetworkfrom gettingtoo“hot”(active)→setpointbehavior
Types ofInhibition Feedback b) a) Hidden Inhib Hidden Inhib Feed− Forward Input Input Anticipatesexcitation Reacts toexcitation
Types ofInhibition Feedback b) a) Hidden Inhib Hidden Inhib Feed− Forward Input Input Anticipatesexcitation Reacts toexcitation Like havingthermostat outside of yourhouse Like a normal(indoor) ACthermostat
<Inhib.proj> parameter explanation Feed-forward inhibition • Strength of FF weights to inhib (ff_wt_scale) • How much the input turns on inhibition Feed-back inhibition • Strength of FB weights to inhib (fb_wt_scale) • Hidden units turning on inhibition Both FF and FB • Inhib conductance into inhib units (g_bar_i.inhib) • Inhibitory units inhibiting each-other • Inhib conductance into hidden units (g_bar_i.hidden) • Inhibitory units inhibiting hidden units
FFFB Inhibition function • Can summarise effects of both types of inhibition by manipulating inhibitory conductance in all units within a layer in a way that would approximate what interneurons would do (while retaining dynamics). • This lets us avoid simulating inhibitory interneurons, which is a big computational saving • Also “cuts to the chase” and avoids oscillations in inhibitory response • In some cases inhibitory neurons may be of interest, and we can still simulate them where desirable. Just don’t need to in every case.
FFFB inhibitionfunction • Wecanapproximatefeedforward(FF)andfeedback(FB)aspectsof inhibitoryinterneuronsusingtheFFFBinhibitionfunction: • averagenetinput:<η>=Ln1ηi n • averageactivation:<y>=Ln1yi n • Then:ff(t)=ff[<η>−ff0]+ • fb(t)=fb(t−1)+dt[fb<y>−fb(t−1)] • NowjustsetgiintargetlayerasafunctionofFFandFB: • gi(t)=gi[ff(t)+fb(t)] Advantages: Much less computationally expensive, avoidsoscillations
Simulations: [inhibfffb.proj] FFFB approximates set pointbehavior. Allowsforfasterupdating,reducesoverallcomputation. Canuseinlargenetworkswithmultiplelayers,withinhibition summarized byFFFB Canstillcapturedifferentialamountsofinhibitionindifferentbrain areas with FFFB params: gi, FF andFB components insomeapplicationsmaystillwantactualinhibneurons
Competition • Inhibition within a layer creates competition • Why might this be useful? • Forces different units to do different things • Turns off noisy (irrelevent) units for a given computation • Onlythemost appropriate(best-fitting)unitssurvivethecompetition
Alternative inhibition function(optional): k-Winners-Take-All(kWTA)
Alternative inhibition function(optional) k-Winners-Take-All(kWTA) • Thefunctionofinhibitionistokeepexcitatoryactivityataroughset point. • Wecanapproximatethisfunctionbyenforcingamaxactivitylevelin eachlayer. • kWTA: Instead of simulating inhibitory neurons, we choose an inhibitorycurrentgi valueforeachlayersuchthatthespecified numberkofexcitatoryneuronsareabovethreshold.
Alternative inhibition function(optional) k-Winners-Take-All(kWTA) • Thefunctionofinhibitionistokeepexcitatoryactivityataroughset point. • Wecanapproximatethisfunctionbyenforcingamaxactivitylevelin eachlayer. • kWTA: Instead of simulating inhibitory neurons, we choose an inhibitorycurrentgi valueforeachlayersuchthatthespecified numberkofexcitatoryneuronsareabovethreshold. • Advantages: Much less computationally expensive, avoidsoscillations.
kWTA:Summary • Simpleshortcutweuseinsteadofactualinhibitoryinterneurons • Capturesbasicideathatinhibitionmaintainsactivityata • set point for a givenlayer • Specifyinhibitionvalueforalayersuchthatkunitsareactive • kisaparameter:percentactivitylevelsvaryacrossdifferentbrain regions! • kWTAstillallowsforsomewiggleroominoverallactivation
Benefits ofInhibition • Controls activity (bidirectionalexcitation) • Inhibitionforcesunitstocompetetorepresenttheinput:Onlythemost appropriate(best-fitting)unitssurvivethecompetition
Networks • Biology: The cortex • Excitation: • Unidirectional(transformations) • Bidirectional (top-down processing, patterncompletion, amplification) • Inhibition: Controls bidirectional excitation (feedforward, feedback,set point, FFFBapproximation) • Constraint Satisfaction: Putting it alltogether.
ConstraintSatisfaction Processoftryingtosatisfyvariousconstraints(fromenvironment, connection weights,activations). Bidirectionalexcitationandinhibitionformpartofthislarger computationalgoal.
ConstraintSatisfaction Processoftryingtosatisfyvariousconstraints(fromenvironment, connection weights,activations). Bidirectionalexcitationandinhibitionformpartofthislarger computationalgoal. Energy/harmony. AttractorDynamics. Noise.
Harmony Harmony=extenttowhichunitactivationsareconsistentwithweights H=1LjLiaiwijaj 2
Harmony Harmony=extenttowhichunitactivationsareconsistentwithweights H=1LjLiaiwijaj 2 Harmonyishighwhenunitswithstrong(positive)weightsareco-active
Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles
Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles
Harmony JohnHopfieldshowedthatharmonytendstoincreasemonotonicallyasthe networksettles networksettling=movingtoamore“harmonious”state
Attractors An attractor network is a network of neurons with excitatory interconnectionsthatcansettleintoastablepatternoffiringgivenarange of different startingstates.
Attractors An attractor network is a network of neurons with excitatory interconnectionsthatcansettleintoastablepatternoffiringgivenarange of different startingstates. [hereweconsideronlyfixedpointattractors,butcyclicalorchaotic attractors are alsopossible]
AttractorDynamics Bidirectionalexcitationcausesanetworktosettleintoaparticularstable state over time: theattractor. Cool Demos on Hopfield attractors: http://jackterwilliger.com/attractor-networks/
AttractorDynamics Bidirectionalexcitationcausesanetworktosettleintoaparticularstable state over time: theattractor. Circle indicates attractor basin. Maximizeharmonygiveninputsandweights.
The NeckerCube a) b) c) • Two differentinterpretations • Can’t perceive both atonce • Alternate between perceptions:bistability
The Role ofNoise Howmightnoisebeusefulinyourbrain?