Neural Networks for Optimization

1. Neural Networks for Optimization Bill Wolfe California State University Channel Islands

3. Neural Models • Simple processing units • Lots of them • Highly interconnected • Exchange excitatory and inhibitory signals • Variety of connection architectures/strengths • “Learning”: changes in connection strengths • “Knowledge”: connection architecture • No central processor: distributed processing

4. Simple Neural Model • aiActivation • ei External input • wij Connection Strength Assume: wij = wji (“symmetric” network) W = (wij) is a symmetric matrix

5. Net Input Vector Format:

6. Dynamics • Basic idea:

7. Energy

9. Lower Energy • da/dt = net = -grad(E)  seeks lower energy

10. Problem: Divergence

11. A Fix: Saturation

12. Keeps the activation vector inside the hypercube boundaries Encourages convergence to corners

13. Summary: The Neural Model aiActivation eiExternal Input wijConnection Strength W (wij = wji) Symmetric

14. Example: Inhibitory Networks • Completely inhibitory • wij = -1 for all i,j • k-winner • Inhibitory Grid • neighborhood inhibition

15. Traveling Salesman Problem • Classic combinatorial optimization problem • Find the shortest “tour” through n cities • n!/2n distinct tours

16. TSP solution for 15,000 cities in Germany

17. TSP50 City Example

18. Random

19. Nearest-City

20. 2-OPT

21. An Effective Heuristic for the Traveling Salesman Problem S. Lin and B. W. Kernighan Operations Research, 1973 http://www.jstor.org/view/0030364x/ap010105/01a00060/0

22. Centroid

23. Monotonic

24. Neural Network Approach neuron

25. Tours – Permutation Matrices tour: CDBA permutation matrices correspond to the “feasible” states.

26. Not Allowed

27. Only one city per time stopOnly one time stop per cityInhibitory rows and columns inhibitory

28. Distance Connections: Inhibit the neighboring cities in proportion to their distances.

29. putting it all together:

30. Research Questions • Which architecture is best? • Does the network produce: • feasible solutions? • high quality solutions? • optimal solutions? • How do the initial activations affect network performance? • Is the network similar to “nearest city” or any other traditional heuristic? • How does the particular city configuration affect network performance? • Is there a better way to understand the nonlinear dynamics?

31. typical state of the network before convergence

32. “Fuzzy Readout”

33. Initial Phase Fuzzy Tour Neural Activations

35. Monotonic Phase Fuzzy Tour Neural Activations

36. Nearest-City Phase Fuzzy Tour Neural Activations

37. Fuzzy Tour Lengths tour length iteration

38. Average Results for n=10 to n=70 cities (50 random runs per n) # cities

39. DEMO 2 Applet by Darrell Long http://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html

40. Conclusions • Neurons stimulate intriguing computational models. • The models are complex, nonlinear, and difficult to analyze. • The interaction of many simple processing units is difficult to visualize. • The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic. • Much work to be done to understand these models.

41. EXTRA SLIDES

42. E = -1/2 { ∑i ∑x ∑j ∑y aix ajy wixjy } = -1/2 { ∑i ∑x ∑y (- d(x,y)) aix ( ai+1y + ai-1y) + ∑i ∑x ∑j (-1/n) aix ajx + ∑i ∑x ∑y (-1/n) aix aiy + ∑i ∑x ∑j ∑y (1/n2) aix ajy }

43. wixjy = 1/n2 - 1/n y = x, j ≠ i; (row) 1/n2 - 1/n y ≠ x, j = i; (column) 1/n2 - 2/n y = x, j = i; (self) 1/n2 - d(x, y) y ≠ x, j = i +1, or j = i - 1. (distance ) 1/n2 j ≠ i-1, i, i+1, and y ≠ x; (global )

44. Brain • Approximately 1010 neurons • Neurons are relatively simple • Approximately 104 fan out • No central processor • Neurons communicate via excitatory and inhibitory signals • Learning is associated with modifications of connection strengths between neurons

45. Fuzzy Tour Lengths tour length iteration

46. Average Results for n=10 to n=70 cities (50 random runs per n) tour length # cities

48. with external input e = 1/2

49. Perfect K-winner Performance: e = k-1/2