An Analysis of the Million Module March Algorithm Applied to the ATRON Robotic Platform

1. An Analysis of the Million Module March Algorithm Applied to the ATRON Robotic Platform • James Phipps

2. The Plan • Intro to Self-Reconfiguring Robots • Intro to Markov Decision Processes • SlidingCube overview • Million Module March overview • Formulating an MDP for SlidingCube • ATRON kinematics • How the MDP works with ATRON • Results and Conclusions

3. What is a self-reconfiguring robot • A robot made up of small, independently moving parts • Capable of changing its shape, possibly even its capabilities • Theoretically homogeneous or heterogeneous

4. Implementations • Molecule • Crystal • Superbot • ATRON Images taken from: Distributed Robotics Lab: http://groups.csail.mit.edu/drl, Polymorphic Robotics Laboratory:http://www.isi.edu/robots/superbot.htm, Maersk Mc-Kinney Moller Institute: http://www.mip.sdu.dk/press/robotsatplay.html

5. Types of self-reconfiguring kinematics • Chain based • Modules are connected in linear fashion • Well suited to non-reconfiguring locomotion (wheels, legs, etc) • Lattice based • Modules are connected in gridlike fashion • Well suited to extreme reconfiguration

6. The motion planning problem • Examining lattice based robots • Motion relies on reconfiguration • Movement instructions must be generated for all modules • Goal is loosely defined • Space is a simple shape • Do not care about exact module location within the space

7. Other Motion PlanningAlgorithms • Distributed planning and control for modular robots with unit- compressible modules • Zack Butler, Daniela Rus, 2003 • “PacMan” • Provided parallel planning and actuation for unit-compressible modules • Generic decentralized control for lattice-based self-reconfigurable robots • Zack Butler, Keith Kotay, Daniela Rus, Kohji Tomita, 2004 • Provided generic rule-based control for a lattice based system with given kinematics

8. Other Motion PlanningAlgorithms • Evolution of Shape-Changing and Self-Repairing Control for the ATRON Self-Reconfigurable Robot • David Christensen, 2006 • Distributed algorithm for ATRON using meta-modules • A new meta-module for controlling large sheets of ATRON modules • David Brandt, David Christensen, 2006 • Proposed a 2D ATRON meta-module sliding square • Demonstrated Million Module March

9. Other Motion PlanningAlgorithms • Self-reconfiguration using directed growth • K. Stoy, R. Nagpal, 2007 • Operates on lattice based robotic systems • Uses multiple bounding boxes to represent goalspace • Supports configuration specification • Uses rule-based motion decisions • Cannot handle obstacles

10. Other Motion PlanningAlgorithms • Automated Design of Adaptive Controllers for Modular Robots using Reinforcement Learning • Paulina Varshavskaya, Leslie Pack Kaelbling, Daniela Rus, 2008 • Uses MDP to distill motion rules • Can be distributed • Cannot handle changing terrain

11. Other Motion PlanningAlgorithms • Useful Metrics for Modular Robot Motion Planning • Amit Pamecha, Imme Ebert-Uphoff, Gregory Chirikjian, 1997 • Algorithm based on a method for calculating the “distance” between two configurations • Assumes that a kinematically optimal path corresponds to cartesian distance between portions of starting and ending configurations • Polynomial runtime

12. Markov Property • The conditional probability for transitioning from this state to another state depends only on the current state. • Results of actions can be predicted from current knowledge • History not needed • Allows a problem to be formulated as a Markov Decision Process • The motion planning problem has this trait.

13. Markov Decision Process • A four-tuple <S,A,T,R> • S - set of all possible states • A - set of all possible actions • T - transition function mapping a state and an action to a resulting state • R - reward function • Well studied reinforcement-learning technique • General solutions exist • Definition requires complete knowledge of S

14. Partially-Observable MDP • Similar four-tuple • S and T are stochastic distributions, based on limited observability of the agent • Has been shown to be hard for PSPACE • Papadimitriou and Tsitsikilis, 1987 • Reduced QSAT to POMDP

15. QSAT Reduction • Boolean Formula in Conjunctive Normal Form • There are n variables, x1 to xn, which are both existential and universal • Will be subscripted with j • There are m clauses, C1 to Cm • Will be subscripted with i

16. QSAT Reduction

17. QSAT Reduction • A policy to navigate to e0 with a cost of 0 exists iff there is a variable assignment which causes the formula to be true • Otherwise, expected cost is 2-n/m

18. SlidingCube

19. Million Module March • Algorithm has sublinear time and space requirements • Uses dynamic programming to solve MDP formulation of motion planning problem • Full observability discussed in a few minutes • Proven experimentally on SlidingCube platform • Concurrency handled by locking routines

20. Million Module March • MDP Formulation • S is a location in space • A is set of kinematically possible actions • T is assumed to match A (no unexpected shifts) • R is -1 for all spaces not in the goal and a value based on height inside • MDP is solved with dynamic programming technique • MDP is re-solved every timestep

21. Million Module March • Collisions avoided by “locking” space at move time • Disconnections avoided with local search • State information stored in a hashtable • Simulator is single threaded, algorithm need not be • Algorithm can plan each module independently, as the MDP is actually over space, not robot state

22. SlidingCube as an MDP • Motion planning for SlidingCube is a MDP • SlidingCube kinematics guarantee that any two connected configurations are reachable from each other • With no obstacles

23. Sliding Square Reachability • Step 1: If there is a vertex of degree one which is not part of Z, perform this step. Otherwise, proceed to Step 2. • Step 1A: If there is a vertex of degree one in the outer border, it may be moved without disconnection to the next open position in the horizontal line, thus joining Z. Following this, return to the beginning. • Step 1B:If there is a vertex of degree one which is not on the inner border, it may be moved without disconnection to a position where it is of degree at least two. This is guaranteed by its position as being an inner border (there must exist a “corner”). Following this, return to the beginnning.

24. SlidingSquare Reachability • Step 2: To get here, there is a maximal cycle with a single connector (see full proof).The selected vertex should be part of one of the vertical edges of the cycle which does not contain the singular connector. Since the selected module is not the connector of the cycle, it may be removed without disconnecting the graph. • Step 2A: If the selected vertex is part of the outer border, it may be moved to the next open position in Z. Following this, return to the beginning. • Step 2B: If the selected vertex is not part of the outer border, then move it within its space such that it is no longer part of the cycle.

25. SlidingCube Reachability • Step 1: Make a tree, rooted in maximal X module • Step 2:If there is a leaf on the outer border, move it to the next position in D. Repeat until out of outer leaves. • Step 3: Pick a leaf at random with an ancestor on the outer border. Move all of this ancestor’s children to other ancestors until it is a leaf. Go back to 2.

26. Implications • As long as the goalspace is big enough, there must be a way to get into it. • There is no move you can make to prevent this • Since we don’t care about optimality, we can plan only our next move • Each module may be safely planned by itself • This only works while non-optimal • As long as an obstacle doesn’t box us in, we can overcome it.

27. The ATRON Module • “Spinning Sphere” • Four connection points on each hemisphere • Capable of infinite rotation around its axis • At least two modules necessary for motion

28. ATRON Modules

29. ATRON differences • ATRON relies on neighbors for motion • Different lattice structure • Cubes connected by edges, not faces • Modules rotate around each other • Only one move type • No guaranteed reachability

30. ATRON implementation • MDP • S - contains additional information about neighbor states, and module axis • A - kinematically possible moves, based on neighbor orientations. • Obstacles are not neighbors! • T - Still certain • R - changes to goal rewards

31. Codebase

32. Codebase changes • Fixed hack which stopped motion in goal • Significant module and kinematic function changes to match ATRONs kinematics • New query interface • Didn’t add axis value to reward function • Was trying to create agnostic implementation

33. ATRON Visualization

34. ATRON Starting States

35. ATRON Starting States

36. Demo

39. Failure Modes • Kinematic Orphaning • Got left at the end of a chain • Stuck in a small orbit • Thrashing • Local maxima in reward values

40. Module Neighborhood

41. Experimental Results

42. Summary • It doesn’t work! • Kinematic platforms similar to SlidingCube have a guaranteed reachability • The key to this is likely the two move types- can traverse our own surface always • Million Module March works because of guaranteed kinematic reachability • This is an MDP over the distance to the goal

43. Summary • ATRON doesn’t have this guarantee • This appears to be a POMDP over the state of the robot, so it cannot be solved in time sublinear to the robot • ATRON appears to be kinematically distinct from SlidingCube • No known meta-module (3d) reduction

44. References • R. Fitch and Z. Butler, “Scalable locomotion for large self-reconfiguring robots,” in Robotics and Automation, 2007 IEEE International Conference on, pp. 2248 –2253, April 2007. • D. Christensen, “Evolution of shape-changing and self-repairing control for the atron self- reconfigurable robot,” in Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, pp. 2539 –2545, May 2006. • D. Brandt and D. Christensen, “A new meta-module for controlling large sheets of atron mod- ules,” in Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Confer- ence on, pp. 2375 –2380, November 2007. • T. Fukuda and S. Nakagawa, “Dynamically reconfigurable robotic system,” in Robotics and Automation, 1988. Proceedings., 1988 IEEE International Conference on, vol. 3, pp. 1581 –1586, April 1988. • B. Salemi, M. Moll, and W.-M. Shen, “Superbot: A deployable, multi-functional, and mod- ular self-reconfigurable robotic system,” in Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 3636 –3641, October 2006. • M. Jorgensen, E. Ostergaard, and H. Lund, “Modular atron: modules for a self-reconfigurable robot,” in Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings. 2004 IEEE/RSJ International Conference on, vol. 2, pp. 2068 – 2073, September-October 2004. • K. Kotay, D. Rus, M. Vona, and C. McGray, “The self-reconfiguring robotic molecule,” in Robotics and Automation, 1998. Proceedings. 1998 IEEE International Conference on, vol. 1, pp. 424 –431, May 1998. • D. Rus and M. Vona, “Self-reconfiguration planning with compressible unit modules,” in Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on, vol. 4, pp. 2513 –2520, 1999.

45. References • R. Fitch and Z. Butler, “Million module march: Scalable locomotion for large self-reconfiguring robots,” in The International Journal of Robotics Research, vol. 27, pp. 331–343, March 2008. • Z. Butler, K. Kotay, D. Rus, and K. Tomita, “Generic decentralized control for lattice-based self-reconfigurable robots,” The International Journal of Robotics Research, vol. 23, pp. 919– 937, September 2004. • Z. Butler and D. Rus, “Distributed planning and control for modular robots with unit- compressible modules,” The International Journal of Robotics Research, vol. 22, pp. 699–715, September 2003. • K. Stoy and R. Nagpal, “Self-reconfiguration using directed growth,” in Distributed Au- tonomous Robotic Systems 6, pp. 3–12, Springer Japan, 2007. • P. Varshavskaya, L. P. Kaelbling, and D. Rus, “Automated design of adaptive controllers for modular robots using reinforcement learning,” The International Journal of Robotics Research, vol. 27, no. 3-4, pp. 505–526, 2008. • A. Pamecha, I. Ebert-Uphoff, and G. Chirikjian, “Useful metrics for modular robot motion planning,” Robotics and Automation, IEEE Transactions on, vol. 13, pp. 531 –545, August 1997. • C. H. Papadimitriou and J. N. Tsitsiklis, “The complexity of markov decision processes,” MATHEMATICS OF OPERATIONS RESEARCH, vol. 12, no. 3, pp. 441–450, 1987. • A. Dumitrescu and J. Pach, “Pushing squares around,” in SCG ’04: Proceedings of the twenti- eth annual symposium on Computational geometry, (New York, NY, USA), pp. 116–123, ACM, 2004.