An Order-Theoretic Approach toward Entropy

1. An Order-Theoretic Approach toward Entropy Kevin H. Knuth Department of Physics University at Albany

2. In the Beginning… Kevin H KnuthFacets of Entropy

3. Lifting Rocks This caveman finds it easy to order these rocks in terms of how heavy they are to lift. Kevin H KnuthFacets of Entropy

4. Diagram Representing the Ordering Heavier He uses a binary weight comparison to order his rocks This configuration is called a chain Kevin H KnuthFacets of Entropy

5. Isomorphisms 5 4 3 Is Less than or Equal to 2 1 16 8 4 Divides Is Less Ripe than 2 1 Kevin H KnuthFacets of Entropy

6. Antichains These elements are incomparable under our binary ordering relation They form a poset called an ANTICHAIN Kevin H KnuthFacets of Entropy

7. Partitioning A set of elements together with a binary ordering relation based onthe notion of containing Kevin H KnuthFacets of Entropy

8. Two Posets with Integers 4 8 3 4 6 9 Is Less than or Equal to Divides 2 5 7 2 3 1 1 Kevin H KnuthFacets of Entropy

9. The Powerset of {a, b, c} Is a subset of Kevin H KnuthFacets of Entropy

10. Posets A partially ordered set (poset) is a set of elements together with a binary ordering relation . includes covers Kevin H KnuthFacets of Entropy

11. Lattices • A lattice is a poset P where every pair of elements x and y has • a least upper bound called the join • a greatest lower bound called the meet The green elements are upper bounds of the blue circled pair. The green circled element is their least upper bound or their join. Similarly Kevin H KnuthFacets of Entropy

12. Lattice Identities The Lattice Identities L1. Idempotent L2. Commutative L3. Associative L4. Absorption If the meet and join follow the Consistency Relations C1. (x is the greatest lower bound of x and y) C2 . (y is the least upper bound of x and y) Kevin H KnuthFacets of Entropy

13. Lattices are Algebras StructuralViewpoint OperationalViewpoint Kevin H KnuthFacets of Entropy

14. Lattices are Algebras StructuralViewpoint OperationalViewpoint Sets, Is a subset of Positive Integers, Divides Assertions, Implies Integers, Is less than or equal to Kevin H KnuthFacets of Entropy

15. Hypothesis Space • (Our States of Knowledge)

16. Describing a State of Knowledge These statements describe the fruit: a = ‘It is an Apple!’ b = ‘It is a Banana!’ c = ‘It is an Citrus Fruit!’ They can also describe what one knows about the fruit. Kevin H KnuthFacets of Entropy

17. The Hypothesis Space The atoms are mutually exclusive and exhaustive logical statements a = ‘It is an apple!’ b = ‘It is a banana!’ c = ‘It is a citrus fruit!’ a b c Kevin H KnuthFacets of Entropy

18. The Boolean Hypothesis Space The meet of any two atoms is the absurdity: a b =  We do not allow our state of knowledge to include: ‘The fruit is an apple AND a banana!’ a b c  Kevin H KnuthFacets of Entropy

19. The Boolean Hypothesis Space a b c  Kevin H KnuthFacets of Entropy

20. The Boolean Hypothesis Space The join of any two elements represents a logical OR: a  b a  b a b c  Kevin H KnuthFacets of Entropy

21. The Boolean Hypothesis Space The join of any two elements represents a logical OR: a  b a  b a  c b  c a b c  Kevin H KnuthFacets of Entropy

22. The Boolean Hypothesis Space The final join gives us the TOP element, often called the TRUISM  = a  b  c “It is an Apple or a Banana or an Orange!”  a  b a  c b  c a b c  Kevin H KnuthFacets of Entropy

23. The Hypothesis Space  a  b a  c b  c a b c  This is aHYPOTHESIS SPACE!!! It consists of all the statements that can be constructed from a set of mutually exclusive exhaustive statements. The space is ordered by the ordering relation “implies” We allow concepts like: ‘The fruit is an apple OR a banana!’ while we disallow concepts like: ‘The fruit is an apple AND a banana!’ Kevin H KnuthFacets of Entropy

24. Superpositions of States??? This was your initial state of knowledge. The fruit was never in this state! This was the state of the fruit.  Kevin H KnuthFacets of Entropy

25. Two Spaces  Space of statements describing the fruit Space of statements describinga state of knowledge about the fruit Kevin H KnuthFacets of Entropy

26. Generalizing Partial Orders to Measures

27. Inclusion and the Zeta Function  a  b a  c b  c a b c  The Zeta function encodes inclusion on the lattice. Kevin H KnuthFacets of Entropy

28. Inclusion and the Zeta Function  a  b a  c b  c a b c since since  The Zeta function encodes inclusion on the lattice. Kevin H KnuthFacets of Entropy

29. The Zeta Function  a  b a  c b  c a b c  Kevin H KnuthFacets of Entropy

30. Inclusion and the Zeta Function The Zeta function encodes inclusion on the lattice. We can define its dual by flipping around the ordering relation Kevin H KnuthFacets of Entropy

31. Degrees of Inclusion and Z We generalize the dual of the Zeta function to the function z Kevin H KnuthFacets of Entropy

32. Z The function z Continues to encode inclusion, but has generalized the concept to degrees of inclusion. In the lattice of logical statements ordered by implies, this function describes degrees of implication. Kevin H KnuthFacets of Entropy

33. How do we Assign Values to z? Are all of the values of the function z arbitrary? Or are there constraints? Here there be monsters… Kevin H KnuthFacets of Entropy

34. Lattice Structure Imposes Constraints I showed that in “general”: Associativity leads to a Sum Rule… Distributivity leads to a Product Rule… Commutivity leads to Bayes Theorem… Kevin H KnuthFacets of Entropy

35. Inclusion-Exclusion (The Sum Rule) The Sum Rule for Lattices Kevin H KnuthFacets of Entropy

36. Inclusion-Exclusion (The Sum Rule) The Sum Rule for Probability Kevin H KnuthFacets of Entropy

37. Inclusion-Exclusion (The Sum Rule) Definition of Mutual Information Kevin H KnuthFacets of Entropy

38. Inclusion-Exclusion (The Sum Rule) Polya’s Min-Max Rule for Integers Kevin H KnuthFacets of Entropy

39. Inclusion-Exclusion (The Sum Rule) This is intimately related to the Möbius function for the lattice, which is related to the Zeta function. Kevin H KnuthFacets of Entropy

40. Probability Changing notation The MEANING of p(x|y) is made explicit via the Zeta function. These are degrees of implication!NOT plausibility!NOT degrees of belief!NOT frequencies of occurrences! Kevin H KnuthFacets of Entropy

41. Statement – QuestionDuality Kevin H KnuthFacets of Entropy

42. Defining a Question Richard T. Cox (1979) defined a question as the set of all possible assertions that answer it. I recast his definition to obtain new insights. In lattice theory, such a set is called a down-set In the figure the colored elementsbelong to I call the set of top elements theirreducible set. Kevin H KnuthFacets of Entropy

43. When are Questions Equal? Two questions areequivalentwhen they are defined by the same set of assertions. Consider the questions “Is it raining?” “Is it not raining?” They are both answered by the set of assertions generated by the irreducible set { “It is raining!”, “It is not raining!”} They are therefore equivalent. Kevin H KnuthFacets of Entropy

44. Animal, Vegetable, Mineral a v m Space of Statementsdescribing a state of knowledge about the object Space of statementsdescribing the object Kevin H KnuthFacets of Entropy

45. The Central Issue I = “Is it an Animal, a Vegetable, or a Mineral?” This question is answered by the following set of statements: I = { a = “It is an animal!”, v = “It is a vegetable!”, m = “It is a mineral!” } Kevin H KnuthFacets of Entropy

46. Some Questions Answer Others As the defining set of I is exhaustive, Now consider the binary question B = “Is it an animal?” B = {a = “It is an animal!”, ~a = “It is not an animal!”} Kevin H KnuthFacets of Entropy

47. Ordering Questions I answers B B includes I I = “Is it an Animal, a Vegetable, or a Mineral?” B = “Is it an animal?” Kevin H KnuthFacets of Entropy

48. Meets and Joins of Questions With “is a subset of” as the ordering relation among questions, one can show that: The meetof two questions is theset intersectionof the set of assertions answering the question. The joinof two questions is theset unionof the set of assertions answering the question. Kevin H KnuthFacets of Entropy

49. Ideals An idealis a nonvoid subset J of a lattice A with the properties (Birkhoff 1967) I1. , where then I2. , then I1 is the condition for the set J to be a down-set, or equivalently a question. I2 assures that there is a unique maximum Therefore, as ideals are questions, I call themIdeal Questions. Kevin H KnuthFacets of Entropy

50. Ideals and Ideal Questions Kevin H KnuthFacets of Entropy