Physics and Machine Learning

1. Physics and Machine Learning “All the tricks that physicists’ use eventually end up in machine learning”

2. Energy – Physics definitions Energy - A measure of being able to do work. There are many forms of energy, such as heat, mechanical, electrical, radiant, chemical, and nuclear energies. Energy is measured in such units as the joule (J), erg, kilowatt-hour (kW-hr), kilocalorie (kcal), foot-pound (ft-lb.), electron-volt (ev), and British thermal unit (BTU). –NASA.gov "It is important to realize that in physics today, we have no knowledge of what energy is. We do not have a picture that energy comes in little blobs of a definite amount." -Richard Feynman "Lectures on Physics"

3. Energy - Physics II • Magnetic Arrays and Spin Images: http://meso.phys.northwestern.edu/research/magneticarrays.html

4. Energy - Machine Learning WiJ • H=-½iJwiJSiSJ Si SJ Vs. Vs. Vs. Vs.

5. Energy - Machine Learning II • Energy is the difference in weight between all nodes that agree and all nodes that disagree. • The more weights, the greater energy. • The “closer” the call, the lower |H|.

6. Energy Minima • Retrieval States – attractors • Mixture States – linear combinations of odd numbered attractors • Spin Glass States – uncorrelated to attractors.

7. Ferromagnetics

8. Energy Metaphor Imagine the atomic magnets as movable objects with the freedom to flip, but you control their position. Each iteration of learning is like forcing all magnets to be closer together, as such the network energy is potential energy and the flipping of spins is the expression of kinetic energy.

9. Temperature – Physics I • Extending the ferromagnetic example • As temperature increases, the impact of other atomic magnets’ spins is decreased. • At absolute zero, temperature has no impact. • At the critical temperature(Tc), spin has no impact.

10. Temperature – Ferromagnetics • Si = +1 w/ probability g(hi); else -1 • g(h) = 1/(1+exp(-2βh)) • β= 1/(kBT) • kB = Boltzman’s constant • T = temperature • Fβ(+/-hi)=1/(1+exp(-/+ 2βhi)Fβ(+/-hi) • Fβ(+/-hi) is a logistic function

11. Temperature – Machine Learning • Logistic function • Noise • Used in the elimination of spurious local minima

12. Mean Field Theory – Physics • The individual measurement and summation of each member of a magnetic array is too expensive • Physicists look to average values as an inexpensive way to extract further truth from a complex combinatronics problem.

13. <S> 1 T Tc Mean Field Theory – Physics • hi=JwiJSJ+hext • <hi>=JwiJ<SJ>+hext • <Si>=tan(β<hi>) = tanh(βJwiJ<SJ>+hext) • <S>=tanh(βJ(S))

14. Mean Field Theory – stochastic model • <Si>=tanh(β/NJuζuiζuJ<SJ>) • We allow an assumption, that <Si> is proportional to one of the stored patterns ζvi • <Si>=m ζvi • <Ncorrect>=½N(1+m) T Tc

15. Mean Field Theory • <Ncorrect>=½N(1+m) • There is a point at which noise overcomes the ability of a network to make an informed decision.

16. Conclusions • All these metaphors that pull from physics are very tightly linked to energy. • All metaphors concentrate on atomic events. (exception that proves the rule: mean field theory).

17. Extra Time? Extra Topics!

18. Entropy – Physics • The inevitable progression toward chaos • The motion of energy and matter away from an organized state.

19. Entropy – Machine Learning • S = -PlogP • For S = -Plog2P (the binary case) this is the average amount of additional information required to specify one of the staties.

20. Quantum Mechanics Fits within the context of our expectation for where to look for Physics crossover • Atomic – discrete and binary • Energy specific Last Class Lecture – use of Dyads.