Identifying Repeated Patterns of Behavior in Time

1. Identifying Repeated Patterns of Behavior in Time Magnus S. Magnusson Research Professor Human Behavior Laboratory University of Iceland www.hbl.hi.is

2. Francis CRICK: “Another key feature of biology is the existence of many identical examples of complex structures.” (1989, p. 138.) (Crick and Watson discovered the double helix structure of DNA.) Some examples: DNA Base pairs, genes, chromosomes, genomes Behavior Words, gestures and patterns of these Clothing Shoes, hats, coats,.. Urban environments Houses, streets, cars, shops, books, radios... Life and Repetitionof Spatio/Temporal Patterns

3. Behavior is Patterns- often hidden patterns “Behavior consists of patterns in time. Investigations of behavior deal with sequences that, in contrast to bodily characteristics, are not always visible.” Opening words of Eibl-Eibesfeldt’s Ethology: The Biology of Behavior, 1970, p. 1; {Emphasis added.}

4. Self-organization –The Emergence of Patterns Visibleor hidden Bénard cells From Scott Kelso, 1997

5. “Emergence” often needs to be assisted • “It is rarely, if ever, the case that the appropriate notion of pattern is extracted from the phenomenon itself using minimally biased procedures. Briefly stated, in the realm of pattern formation ‘patterns’ are guessed and then verified.” Crutchfiled, J., 1993. (Here cited from Solé & Goodwin, 2000, p. 20).

6. New Research Directions • Much theoretical and methodological thinking within the behavioral sciences (and statistics) stems from the time before cheap powerful computers and advanced software development tools • Highly complex search patterns and algorithms can now be developed and applied • Behavioral scientists can aim for new kinds of discoveries

7. Basic Viewpoint and Task • Behavior is more structured than is perceived directly or through standard data analysis methods • To fully disclose its structure new pattern types and detection methods are needed to complement existing ones • The discovery of hidden patterns is of considerable importance for theoretical and practical reasons

8. Architechture vs. Structurea simple philosophy • Search algorithms should correspond to the structure of the phenomenon being studied • Even the most sophisticated and powerful square detection algorithm is not adequate for the detection of planetary orbits • An imperfect ellipse detection algorithm would be preferable

9. Some Basic Questions • What kinds of hidden significant structure exists in behavior? • How to characterize such structure? • How to discover such hidden structure? • How to discover effects of independent variables on such structure?

10. A sequence: “1. an arrangement of two or more things in successive order” “3. an action or event that follows another or others” “Maths. a. an ordered set of numbers or other mathematical entities in one-to-one correspondence with the integers 1 to n” - Collins. A pattern: “1. an arrangement of repeated or corresponding parts, decorative motives etc...” “Most mathematicians defineMathematics as the science of patterns….” Sequences and Patterns A pattern (shape, form) may not be a sequence but may still include one. Detecting a pattern may thus mean detecting a sequence.

11. Behavior as Repeated Patterns • Linguistics: repeated hierarchical/syntactic patterns • Ethology: repeated hierarchical/syntactic patterns • Behaviorism: repeated real-time contingencies • Anthropology, Social Psychology and more: scripts, plans, routines, strategies, rituals, ceremonies, etc. • The importance of repeated patterns in behavior is widely accepted • The recognition of the “hiddenness” of some such patterns is needed • New pattern definitions with corresponding detection algorithms and tools (software) are required

12. Verbal and Nonverbal are One • “The activity of man constitutes a structural whole, in such a way that it cannot be subdivided into neat “parts” or “levels” or “compartments” insulated in character, content, and organization from other behavior.Verbal and nonverbal activity is a unified whole,andtheory and methodology should be organized or created to treat it as such.” Pike(1960, p. 2).{Emphasis added.}

13. Repetition versus Uniqueness “.. a conversation, … a complex system of relationships which nonetheless may be understood in terms of general principles which are discoverable and generally applicable, even though the course of any specific encounter is unique (cf. Kendon 1963, Argyle and Kendon 1967).” (Kendon, 1990, p. 4). (Emphasis added.)

14. Towards a General Pattern TypeDifferent Time Scales and Content These patterns of patterns have aspects in common: • How do you do? • How do you do? Very well, thank you. • Pass me the salt, Jack. Jack, passes the salt. • If..then..else • Dinner: Sit down..take an entrée..take a main course..take dessert..drink coffee..stand up • Rituals, ceremonies, routines, genes, poems, hospital operations, conferences, classes, football matches, strikes, and melodies

15. Common Structural Aspects • Fixed order and “significantly invariant distances” between components • Hierarchical/syntactic structure • Self similarity / scale independence

16. Such Patterns are Often Hard to Spot dek w akb cdk w kdewkakb ckdw dekwakbcdk w kdewkakbckdw d ek w akb c d k w k d e wkakb ckd w 1 T d ek w akb c d k w k d e wkakb ckd w 2 T 1 and 2 are identical data sets

20. A Dinner...a Time-Flexible Pattern of Patterns • The six sub-patterns are separated by relatively freely structured intervals of relatively fixed length. (From Magnusson, 2005.)

21. Patterns and Causation • Very well, thank you • an earlier word is usually not considered as a cause of any word following it within such intra-individualpatterns • How do you do? Very well, thank you • an earlier part of some inter-individualpatterns may be seen as a likely cause of a later part of the same pattern

22. The T-patterns • A t-pattern is a particular set of event-types recurring in the sameorder (and/or concurrently) with “significantly similardistances” between them on a single dimension. • T-patterns have a scale-independent and hierarchical structure (often syntactically constrained -- “grammar”) • T-patterns may occur randomly, but they often occur in cycles and even when their elements do not

23. The Data: Series of Points on 1-Dim. • An event-type may be an actor’s (agent’s) beginning or ending of a particular behavior. • It may also be a base in a DNA molecule or an amino (recid.) in a protein. a a a a a a Sets of such series form multivariate point series to which all T-pattern definitions refer exclusively.

24. Multivariate Point SeriesThe Basic Data Type A . . . . . . . . .. B .. . . . . C . . .. . D . . E . . . . . . .... . . . .. F . . G . .. . . . . . _______________________________________ t1 t2

25. Recursive T-Pattern Definition with the event-type as the simplest T-pattern The T-pattern is an ordered set of T-patterns (X): X1≈dt1 X2≈dt2 .. Xi-1≈dti-1 Xi.. ≈dtm-1 Xm that recurs with significantly similar time distances, ≈dt,between its elements relative to the zero hypothesis (fiction) of constant probability per unit time for each Xi = NXi / observation time

26. Towards a Detection AlgorithmSearching for Critical Intervals [d1, d2] Comparing SeriesA and B Detected Critical (distance) Interval (window) Repeatedly, an A is followed by a B within approximately the same distance A B d1 d2

27. Critical Intervals and Binary Trees • Any T-pattern Q = X1 X2..Xm can be split into a pair of shorter ones related by a critical interval:QLeft [d1, d2] QRight • Recursively, QLeftand QRight can each be split until the whole pattern X1..Xm is expressed as the terminals of a binary-tree

28. Bottom-up Detection Patterns Grow & Compete • The bottom-up algorithm detects patterns graduallyfrom event types, as pairs of pairs, i.e., as binary trees • It detects critical interval relations between the occurrence series of event types and/or already detected patterns and thenconnects these to form longer patterns (trees). • Many binary-trees may correspond fully or partly to the same pattern so all detected patterns are automatically compared and only the most complete (longest) patterns are kept.

29. Completeness Competition -partial and equivalent trees • (( A B) ((C D) (E F))) • (( A ((B C) D)) (E F)) • (( A B) (D (E F )) • (( A C ) F ) • (B E) • (B D) • (A F)

30. Behavior Record - ExampleTwo Children (Blue & Red) Play With One Toy for 13.5 Min; 81 Series Event Type Occurrence Series Time in 1/15 s

31. New Pattern Presentationfor Complex Patterns

34. Nature’s Symmetries are Approximate • Different instances of the same t-pattern may have quite different internal intervals • Still the relationship is always the same, the Critical Interval Relationship. • Should this be called relative translation symmetry?

35. Statistical Validation

36. Statistical Validation Types

37. Standard Statistical MethodsInadequate for T-pattern Detection • Multivariate statistical methods: look for clouds of points in n-dimensional space rather than for syntactic structures on one dimension • Time series analysis looks for trends or waves rather than hierarchical discontinuities occurring irregularly • Sequential analysis may look for a priori unlikely time sequences but involves no concept of complex repeated 1-D shapes or patterns. It may therefore detect a multitude of sequential relations without ever detecting such underlying patterns

38. 25 min of Children’s Dyadic Problem Solving Data from published studies by Beaudichon and Magnusson.

40. Univariate Bursts

41. Doctor-patient facial interaction data coded with FACS

42. Interindividual T-patterns in Doctor-Patient Facial Interactions Data from the Psychiatric Hopitals in Geneva, V. Haynal et al.

43. Wild-type Male vs. wild-type Mated Female Wild-type Male vs. Mature wild-type Virgin Female #P = 3 #P = 4 Wild-type Male vs. Immature wild-type Female Drosophila interactions B. Arthur’s data and patterns #P = 8

44. The T-System or T-modelA System of Mathematically Defined Terms • The t-patterntypeis the basis of a growing system of terms for the description of temporal structure in complex behavioral processes • Corresponding detection algorithms have been developed and implemented in the THEME software • See patternvision.com & noldus.com

45. Extending the T-modelBuilding on the Critical Interval and T-pattern Concepts • Markers & Indicators • Composition • +/- Associates; Satellites & taboos • Gravity- and repulsion zones • Packets • Packet markers • Drifters • T-kappa

46. T-markers • A t-marker of a t-pattern occurs almost exclusively as a part of that pattern • A t-marker’s occurrence thus indicates that a particular t-pattern is occurring • A marker that occurs early in a pattern predicts the rest of the pattern • A marker occurring late in the pattern retrodicts the earlier part of the pattern

47. A positive or negative associate of a T-pattern is: some behavior that is not a part of that pattern, but occurs within or around its occurrences significantly more (or less) often than expected by chance Associates may occur only, always, sometimesor even neverwithin or near their corresponding T-pattern The “only and always” case is called at-satellite The (almost) never case is called a t-taboo T-associates

48. The T-packet StructureA T-pattern with its Associates and Zones • An instance of a t-packet showing two t-associate instances • The gravity zone, [ t1, t2 ], of a t-pattern extends from the earliest to the last positive associate • The negative gravity or repulsion zone (not shown) is similarly the interval within which any behaviors tend not to occur • T-packets are simultaneously sequential and non-sequential structures

49. Neurones as Interacting and Networking (Social) Organisms

50. Neuronal T-data Including Breathing(from Nicol, Kendrick, Magnusson, 2005)