BrainGene : computational creativity algorithm that invents novel names

1. BrainGene: computational creativity algorithm thatinvents novel names Maciej Pilichowski & Włodzisław Duch Department of Informatics, Nicolaus Copernicus University, Toruń, Poland School of Computer Engineering, Nanyang Technological University, Singapore Google: W. Duch SSCI 2013, Singapore

2. Most mysterious … Intuition is relatively easy … What features of our brain/minds are the most mysterious? • Consciousness? • Imagination? • Emotions, feelings? • Thinking? Masao Ito, neuroscientist, director of the RIKEN Brain Science Institute, answered: Creativity.

3. Creativity in psychology MIT Encyclopedia of Cognitive Sciences (2001) has 1100 pages,6 chapters about logics, over 100 references to logics in the index. Creativity is covered on 1 page (+1 page about „creative person”). Intuition: 0, not even mentioned in the index. In everyday life we use intuition and creativity more often than logics.It is a necessary part of human-level intelligence. Computational approach to creativity is slowly getting more popular, see http://computationalcreativity.net/, ICCC conferences “Computational Creativity is the study and simulation, by computational means, of behaviour, natural and artificial, which would, if observed in humans, be deemed creative.”

4. Creativity in math H. Poincare, The Foundations of Science(1908): • Mathematical intuition and creativity is a discrimination between promising and useless ideas and their combinations. • Mathematical thinking may be based on heuristic search among sufficiently rich representations. • Math intuition is an interplay between spatial imagination, abstraction and approximate reasoning, and analytical reasoning or visual-spatial and linguistic thinking. This is observed in fMRI neuroimaging (ex. S. Dehaene, 1999). J. Hadamard, An Essay on The Psychology of Invention in the Mathematical Field. Dover Publications, 1949, unconscious sources.

5. Mechanizing inventions J. Hadamard, An Essay on The Psychology of Invention in the Mathematical Field. Dover Publications, 1949 G. Altshuller, Algorithm of Invention. MoscowskiyRabochy, 1969. S. Savransky, Engineering of Creativity. Introduction to TRIZ Methodology of Inventive Problem Solving. CRC Press, 2000.TRIZ: "a problem-solving, analysis and forecasting tool derived from the study of patterns of invention in the global patent literature”. • SIT (systematic inventive thinking) • ASIT (advanced systematic inventive thinking) • USIT (unified structured inventive thinking) • TRIZICS (systematic application of TRIZ)

6. Creativity in business J. Goldenberg, D. Mazursky, S. Solomon, Templates in creative sparks.Science, vol. 285, no. 5433, 1999. J. Goldenberg, D. Mazursky, Creativity in product innovation. Cambridge University Press, 2002 Creativity templates: Attribute dependence, Replacement and Displacement templates. This approach is used for product design by large companies, ex. Philips, Ford, Kodak, Coca-Cola, Motorola ...

7. Some history Koza, J.R.; Keane, M.A.; Streeter, M.J.; Mydlowec, W.; Yu, J.; & Lanza, G. (2003). Genetic Programming IV: Routine Human-Competitive Machine Intelligence, Springer. GECCO Conference offered “Hummies” cash awards for human-competitive results that are patentable as new inventions. Discovery of scientific laws: J.M. Zytkow, H.A. Simon, A theory of historical discovery. Machine Learning, 1, 107, 1986. B. Goertzel, From complexity to creativity. Emergent Patterns and Self-Organizing Dynamics in Mental, Computational and Physical Systems. Springer, 1997.

8. Creativity and the brain • A. Dijksterhuis, T. Meurs, Where creativity resides: The generative power of unconscious thought, Consciousness and Cognition 15(1), 135–146, 2006. • W. Duch, Computational creativity, in: World Congress on Computational Intelligence, Vancouver 2006, 1162–1169. • Duch W, Pilichowski M, Experiments with computational creativity. Neural Information Processing – Letters and Reviews, 11(4-6), 123-133, 2007. • W. Duch, Intuition, insight, imagination and creativity. IEEE Computational Intelligence Magazine, 2(3), 40–52, 2008. • W. Duch, P. Matykiewicz, J. Pestian, Neurolinguisticapproach to natural language processing with applications to medical text analysis.” Neural Networks, vol. 21(10), 1500–1510, 2008.

9. Psychology of creativity G. Wallas, The art of thought (1926): four-stage Gestalt model of problem solving. 4 stages: preparation, incubation, illumination and verification. These stages have been identified in creative problem solving by individuals and small groups of people; additional stages may be added: finding or noticing a problem, proposing interesting questions, frustration period preceding illumination, communication following verification etc. Understanding details of such stages and sequences yielding creative productions is a central issue for creativity research, but it is not sufficient for computational creativity. What we need is approximation to creative processes in the brain.

10. Words in the brain Psycholinguistic experiments show that most likely categorical, phonological representations are used, not the acoustic input. Acoustic signal => phoneme => words => semantic concepts. Phonological processing precedes semantic by 90 ms (N200 ERPs). F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press. Action-perception networks inferred from ERP and fMRI Phonological neighborhood density = the number of words that sound similar to a target word. Similar patterns of brain activations. Semantic neighborhood density = the number of words that are similar in meaning to a target word.

11. Symbols in the brain How do words that we hear, see or are thinking of, activate the brain? Seeing words: orthography => phonology => articulation => semantics. Lateral inferotemporal multimodal area (LIMA) reacts to auditory + visual stimulation, has cross-modal phonemic and lexical links. Adjacent visual word form area (VWFA) in the left occipitotemporal sulcus is unimodal. Perhaps auditory word form area also will be found in the left anterior superior temporal sulcus. Left hemisphere: precise representations of symbols, including phonological components. Right hemisphere probably sees clusters of concepts.

12. Models of creativity 1 Simplest creativity: invention of novel words, understanding of neologisms. Go to the lower level … Look at combinations of phonemes, emergence of morphemes, etc. Creativity = space + priming + imagination + filtering. Space::neural tissue providing space for almost infinite # of distinguishable activation patterns; in the model trained neural networks provide search space of internal states that reflect past experience of the system, constraining possible neural activations;

13. Models of creativity 2 • Priming: reading/hearing/seeing primes the brain; priming of neural networks due to the recent history of activations increases probability of cooperation between neurons encoding fragments of distributed representations. • Imagination:some novel words come to my mind. Many chains of phonemes activate in parallel both words and non-words representations, depending on the strength of synaptic connections. • D. Simonton, “Creative thought as blind-variation and selective retention: Combinatorial models of exceptional creativity,” Physics of Life Reviews, vol. 7, p. 156–179, 2010. • BVSR was invented by D. Campbell (1960), with blind variation linking partial activations of primed neural circuits into chunks.

14. Models of creativity 3 • Filtering: the most active chunks, relevant in a given context, win the competition filtering out less active chunks; • Solution: results appear as spontaneous thoughts, solutions to the problem, intentions for the next action, associations, emotions. chess board domino n o m black white i d o phonological reps

15. The Task Given a description of some object (product, internet site, service, organization, invention) create a neologism that is easy to remember, word reminding us of some qualities strongly associated with the object itself. A real letter from a friend: I am looking for a word that would capture the following qualities: portal to new worlds of imagination and creativity, a place where visitors embark on a journey discovering their inner selves, awakening the Peter Pan within. A place where we can travel through time and space (from the origin to the future and back). So, its about time, about space, infinite possibilities. FAST!!! I need it sooooooooooooooooooooooon.

16. BrainGene preparation Preparation phase: • count the number of occurrences of every word w ∈ W in a large corpus; • split words into n-grams; • put occurrence counts in the matrix at the position determined by n-grams – it will tell us how likely are different n-grams; • normalize word matrix. This matrix describes lexical knowledge network, the space for general word imagery.

17. BrainGene priming Priming phase creates primed knowledge network: • From the description of the product extract initial keywords for phonological priming; • spread the activation to concepts and word-forms that are strongly related (go, goes, went) … this forms a priming set; • count occurrences of every word w ∈ W in this set; • split words into n-grams; • put occurrence counts in the matrix at positions determined by n-grams; • add words for negative priming (inhibit some words); • normalize priming word matrix; • add it to the general word matrix to form primed word matrix.

18. BrainGene filtering Assumption: priming increases probability of well-established associations in an additive way. • Create new words from combinations of the most active n-grams in the final primed word matrix. • Rank novel words using normalized counts for n-grams (phonological plausibility). • Words should not resemble closely any word in the dictionary (including substrings, potentially offending words in various languages, use the multilingual.sensegates.com service). • Add novel words to the result set, retain only highest-ranking These novel words have overlapping n-grams with priming ones.

19. BrainGene workflow Primed knowledge network, rather complex probabilistic model, including various linguistic peculiarities.

20. Some experiments Search for good name for e-book reader (Kindle?): • Priming set (after some stemming): Acquir, collect, gather, air, light, lighter, lightest, paper, pocket, portable, anyplace, anytime, anywhere, cable, detach, global, globe, go, went, gone, going, goes, goer, journey, move, moving, network, remote, road, roads, travel, wire, world, book, data, informati, knowledge, librar, memor, news, word, words, comfort, easi, easy, gentl, human, natural, personal, computer, electronic, discover, educat, learn, read, reads, reading, explor. • Exclusion list (for inhibition): aird, airin, airs, bookie, collectic, collectiv, globali, globed, papere, papering, pocketf, travelog.

21. Results for novel words Created word Word count and # domains in Google • librazone 968 1 • inforizine -- -- • librable 188 -- • bookists 216 -- • inforld 30 -- • newsests 3 -- • memorld 78 1 • goinews 31 -- • libravel 972 -- • rearnews 8 -- • booktion 49 -- • newravel 7 -- • lighbooks 1 -- + popular infooks, inforion, datnews, infonews, journics

22. Words: experiments Portal to new worlds of imagination and creativity… • creatival(creativity, portal), used in creatival.com • creativery(creativity, discovery), creativery.com • discoverity = {disc, disco, discover, discovery, verity, creativity) • digventure={dig, digital, venture, adventure} was new … • imativity(imagination, creativity); • infinitime(infinitive, time) • infinition(infinitive, imagination), already a company name • portravel(portal, travel); sportal (space, sport, portal), taken • timagination(time, imagination); timativity (time, creativity) • tivery(time, discovery); trime (travel, time)

23. Evaluation Large number of experiments performed. In most cases 50-80% of invented names are in use as company names, site names, or product names. Ex. for e-books: infoworld, inforion, bookist, boomation, bookstion, cablects, cablector, dataction, datamation, datnews, datmation, easnews, educatics, electroad, explobal, goinmation, gonewsy, infordata, inforld, inforlds, inforion, infornews, infortion, infonews, inforvel, infravel, newsion, newstion, papnews, travelation, travelnews, wentnews ... In February 2013 some of these internet domains were for sale, for example cablead, easmation, educatnews, infovel, pocketnews, wortion, with prices exceeding 1000 USD. http://domomark.com/ service evaluates how good are domain names, assigning scores in rather naive way.

24. Insights and brains Brain solving problems that required insight vs. analytic way. E.M. Bowden et al., New approaches to demystifying insight TCS 2005. 300 ms before insight a burst of gamma activity was observed in the right anterior superior temporal gyrus (RH-aSTG) , interpreted by the authors as „making connections across distantly related information during comprehension ... that allow them to see connections that previously eluded them”.

25. Conclusions & hopes Creativity is not so mysterious, intuition, insight, creativity in limited domain can be simulated at the human competence level, opening a new vista in creativity research and suggesting new neuroimaging experiments. Even drastically simplified representations of lexical and semantic knowledge are useful for applications in word games, query precisiation, medical applications, common sense. Brain processes analyzed at different levels are great inspiration! Fruitful approach: approximations to knowledge reps in brain networks: memory types/interactions, a priori knowledge, spreading activation in networks, simulations of real brain functions, graphs of consistent concepts, ontology-based enhancements, connection to vector models, etc. Approximate brain neurodynamics!

26. Thank youfor lending your ears ... Google: W. Duch => Papers/presentations/projects