The Living Thing / Notebooks :


What’s so special about speech anyway?

Sam Kriss calls the spamularity the language of god. See also Feral, Thomas Urquhart, natural language processing.

“They’re using phrase-structure grammar, long-distance dependencies. FLN recursion, at least four levels deep and I see no reason why it won’t go deeper with continued contact. […] It doesn’t have a clue what I’m saying.”


“It doesn’t even have a clue what it’s saying back,” she added.

—Peter Watts

Dan Stowell summarises a neural basis for recursive syntax:

For decades, Noam Chomsky and colleagues have famously been developing and advocating a “minimalist” (BTCB14) idea about the machinery our brain uses to process language. […] They propose that not much machinery is needed, and one of the key components is a “merge” operation that the brain uses in composing and decomposing grammatical structures.

Then yesterday I was reading this introduction to embeddings in artificial neural network and NLP, and I read the following:

“Models like [this] are powerful, but they have an unfortunate limitation: they can only have a fixed number of inputs. We can overcome this by adding an association module, A, which will take two word or phrase representations and merge them.” (Bott11)


Arimura, H., Shinohara, T., & Otsuki, S. (1994) Finding Minimal Generalizations for Unions of Pattern Languages and Its Application to Inductive Inference from Positive Data. In In Proc. the 11th STACS, LNCS 775 (pp. 649–660). Springer-Verlag
Autebert, J.-M., Berstel, J., & Boasson, L. (1997) Context-free languages and pushdown automata. In G. Rozenberg & A. Salomaa (Eds.), Handbook of formal languages, vol. 1 (pp. 111–174). New York, NY, USA: Springer-Verlag New York, Inc.
Berstel, J., & Boasson, L. (1990) Transductions and context-free languages. In J. van Leeuwen, A. R. Meyer, M. Nivat, M. Paterson, & D. Perrin (Eds.), Handbook of Theoretical Computer Science, Vol. A: Algorithms and Complexity (pp. 1–278).
Berwick, R. C., Okanoya, K., Beckers, G. J. L., & Bolhuis, J. J.(2011) Songs to syntax: the linguistics of birdsong. Trends in Cognitive Sciences, 15(3), 113–121. DOI.
Bolhuis, J. J., Tattersall, I., Chomsky, N., & Berwick, R. C.(2014) How Could Language Have Evolved?. PLoS Biol, 12(8), e1001934. DOI.
Bottou, L. (2011) From Machine Learning to Machine Reasoning. arXiv:1102.1808 [cs].
Bressler, S. L., & Kelso, J. A. S.(2001) Cortical coordination dynamics and cognition. Trends in Cognitive Sciences, 5(1), 26–36. DOI.
Cancho, R. F. i, & Solé, R. V.(2003) Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences, 100(3), 788–791. DOI.
Chomsky, N. (1969) Aspects of the Theory of Syntax. . The MIT Press
Christiansen, M. H., & Chater, N. (2008) Language as shaped by the brain. Behavioral and Brain Sciences, 31, 489–509. DOI.
Christiansen, M. H., Dale, R. A. C., Ellefson, M. R., & Conway, C. M.(2002) The role of sequential learning in language evolution: computational and experimental studies. In A. Cangelosi & D. Parisi (Eds.), Simulating the Evolution of Language. Springer-Verlag New York, Inc.
Deacon, T. W.(2010) A role for relaxed selection in the evolution of the language capacity. Proceedings of the National Academy of Sciences, 107, 9000–9006. DOI.
Dominey, P. F.(2005) From Sensorimotor Sequence to Grammatical Construction: Evidence from Simulation and Neurophysiology. Adaptive Behavior, 13, 347–361. DOI.
Elman, J. L.(1991) Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7, 195–225. DOI.
Elman, J. L.(1993) Learning and development in neural networks: the importance of starting small. Cognition, 48, 71–99. DOI.
Elman, J. L.(1995) Language as a dynamical system. , 195.
Elman, J. L.(2003) Generalization from Sparse Input. In Proceedings of the 38th Annual Meeting of the Chicago Linguistic Society. Citeseer
Elman, J. L., Bates, E. A., Johnson, M. H., Karmiloff-Smith, A., Parisi, D., & Plunkett, K. (1997) Rethinking Innateness: A Connectionist Perspective on Development (Neural Networks and Connectionist Modeling). . The MIT Press
Gibson, E. (1998) Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1), 1–76.
Greibach, S. A.(1966) The Unsolvability of the Recognition of Linear Context-Free Languages. J. ACM, 13(4), 582–587. DOI.
Greibach, S. A.(1969) An Infinite Hierarchy of Context-Free Languages. J. ACM, 16(1), 91–106. DOI.
Jin, D. Z.(2009) Generating variable birdsong syllable sequences with branching chain networks in avian premotor nucleus HVC. Physical Review E, 80(5), 051902. DOI.
Jin, D. Z., & Kozhevnikov, A. A.(2011) A compact statistical model of the song syntax in Bengalese finch. PLoS Comput Biol, 7(3), -1001108. DOI.
John W Backus. (1959) The syntax and semantics of the proposed international algebraic language of the zurich acmgamm conference. In Proceedings of the International Conference on Information Processing. Zürich: UNESCO
Katz, J., & Pesetsky, D. (2009) The recursive syntax and prosody of tonal music. Ms., Massachusetts Institute of Technology.
Kirby, S. (1998) Learning, Bottlenecks and the Evolution of Recursive Syntax. . Presented at the In E. Briscoe (Ed.), Linguistic
Kirby, S. (2003) Language Evolution. . Oxford University Press, USA
Koshiba, T., Mäkinen, E., & Takada, Y. (1997) Inferring Pure Context-Free Languages from Positive Data. ACTA CYBERNETICA, 14, 469–477.
Kuich, W. (1970) On the entropy of context-free languages. Information and Control, 16(2), 173–200. DOI.
Manning, C. D.(2002) Probabilistic syntax. In Probabilistic linguistics (pp. 289–341). Cambridge, MA: MIT Press
Mcclelland, J. L., Botvinick, M. M., Noelle, D. C., Plaut, D. C., Rogers, T. T., Seidenberg, M. S., & Smith, L. B.(2010) Letting structure emerge: connectionist and dynamical systems approaches to cognition. Trends Cogn Sci, 14(8), 348–356. DOI.
Pereira, F. (2000) Formal grammar and information theory: together again?. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 358(1769), 1239–1253. DOI.
Petersson, K.-M., Folia, V., & Hagoort, P. (2012) What artificial grammar learning reveals about the neurobiology of syntax. Brain and Language, 120(2), 83–95. DOI.
Pullum, G. K., & Gazdar, G. (1982) Natural Languages and Context-Free Languages. Linguistics and Philosophy, 4(4), 471–504.
Shieber, S. M.(1987) Evidence Against the Context-Freeness of Natural Language. In W. J. Savitch, E. Bach, W. Marsh, & G. Safran-Naveh (Eds.), The Formal Complexity of Natural Language (pp. 320–334). Springer Netherlands
Smith, K. (2003) The Transmission of Language: models of biological and cultural evolution.
Smith, K., & Kirby, S. (2008) Cultural evolution: implications for understanding the human language faculty and its evolution. Philosophical Transactions of the Royal Society B: Biological Sciences, 363, 3591–3603. DOI.
Wolff, J. G.(2000) Syntax, parsing and production of natural language in a framework of information compression by multiple alignment, unification and search. Journal of Universal Computer Science, 6(8), 781–829.