The Living Thing / Notebooks :

Generative deep learning models

automatic fractal Trump

Many neural networks, — e.g. adversarial ones and variational ones — are generative - they simulate examples of the phenomenon you wish to analyse. This is close to the simulation-based inference we know about from many Bayesian methods; but substantially broader and weirder.

This is interesting to me for a bunch of reasons. I mostly care about it for making generative art with neural networks, but there is some theory here that isn’t about fractal Donald Trumps or whatever.



Arora, S., Ge, R., Liang, Y., Ma, T., & Zhang, Y. (2017) Generalization and Equilibrium in Generative Adversarial Nets (GANs). ArXiv:1703.00573 [Cs].
Bahadori, M. T., Chalupka, K., Choi, E., Chen, R., Stewart, W. F., & Sun, J. (2017) Neural Causal Regularization under the Independence of Mechanisms Assumption. ArXiv:1702.02604 [Cs, Stat].
Blaauw, M., & Bonada, J. (2017) A Neural Parametric Singing Synthesizer. ArXiv:1704.03809 [Cs].
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2015) Generating Sentences from a Continuous Space. ArXiv:1511.06349 [Cs].
Chen, X., Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 29 (pp. 2172–2180). Curran Associates, Inc.
Collins, N. (2008) The Analysis of Generative Music Programs. Organised Sound, 13(03), 237–248. DOI.
Denton, E., Chintala, S., Szlam, A., & Fergus, R. (2015) Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. ArXiv:1506.05751 [Cs].
Dosovitskiy, A., Springenberg, J. T., Tatarchenko, M., & Brox, T. (2014) Learning to Generate Chairs, Tables and Cars with Convolutional Networks. ArXiv:1411.5928 [Cs].
Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017) Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. ArXiv:1704.01279 [Cs].
Fraccaro, M., Sø nderby, S. ren K., Paquet, U., & Winther, O. (2016) Sequential Neural Models with Stochastic Layers. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 29 (pp. 2199–2207). Curran Associates, Inc.
Gal, Y., & Ghahramani, Z. (2015) On Modern Deep Learning and Variational Inference. In Advances in Approximate Bayesian Inference workshop, NIPS.
Gal, Y., & Ghahramani, Z. (2016) Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference. In 4th International Conference on Learning Representations (ICLR) workshop track.
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) Explaining and Harnessing Adversarial Examples. ArXiv:1412.6572 [Cs, Stat].
Graves, A. (2013) Generating Sequences With Recurrent Neural Networks. ArXiv:1308.0850 [Cs].
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) DRAW: A Recurrent Neural Network For Image Generation. ArXiv:1502.04623 [Cs].
He, K., Wang, Y., & Hopcroft, J. (2016) A Powerful Generative Model Using Random Weights for the Deep Image Representation. ArXiv:1606.04801 [Cs].
Hinton, G. E.(2007) Learning multiple layers of representation. Trends in Cognitive Sciences, 11(10), 428–434. DOI.
Hinton, G., Osindero, S., & Teh, Y. (2006) A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), 1527–1554. DOI.
Jetchev, N., Bergmann, U., & Vollgraf, R. (2016) Texture Synthesis with Spatial Generative Adversarial Networks. In Advances in Neural Information Processing Systems 29.
Karpathy, A., & Fei-Fei, L. (2014) Deep Visual-Semantic Alignments for Generating Image Descriptions. ArXiv:1412.2306 [Cs].
Krishnan, R. G., Shalit, U., & Sontag, D. (2016) Structured Inference Networks for Nonlinear State Space Models. ArXiv:1609.09869 [Cs, Stat].
Kulkarni, T. D., Whitney, W., Kohli, P., & Tenenbaum, J. B.(2015) Deep Convolutional Inverse Graphics Network. ArXiv:1503.03167 [Cs].
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y.(n.d.) Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations. . Presented at the Proceedings of the 26th International Confer- ence on Machine Learning, 2009
Leskovec, J., Chakrabarti, D., Kleinberg, J., Faloutsos, C., & Ghahramani, Z. (2010) Kronecker Graphs: An Approach to Modeling Networks. J. Mach. Learn. Res., 11, 985–1042.
Louizos, C., & Welling, M. (2016) Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors. ArXiv Preprint ArXiv:1603.04733.
Mnih, A., & Gregor, K. (2014) Neural Variational Inference and Learning in Belief Networks. In Proceedings of The 31st International Conference on Machine Learning.
Mohamed, A. r, Dahl, G. E., & Hinton, G. (2012) Acoustic Modeling Using Deep Belief Networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), 14–22. DOI.
Paul, A., & Venkatasubramanian, S. (2014) Why does Deep Learning work? - A perspective from Group Theory. ArXiv:1412.6621 [Cs, Stat].
Poole, B., Alemi, A. A., Sohl-Dickstein, J., & Angelova, A. (2016) Improved generator objectives for GANs. In Advances in Neural Information Processing Systems 29.
Radford, A., Metz, L., & Chintala, S. (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ArXiv:1511.06434 [Cs].
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ArXiv:1401.4082 [Cs, Stat].
Salakhutdinov, R. (2015) Learning Deep Generative Models. Annual Review of Statistics and Its Application, 2(1), 361–385. DOI.
Sun, Z., Liu, J., Zhang, Z., Chen, J., Huo, Z., Lee, C. H., & Zhang, X. (2016) Composing Music with Grammar Argumented Neural Networks and Note-Level Encoding. ArXiv:1611.05416 [Cs].
Theis, L., & Bethge, M. (2015) Generative Image Modeling Using Spatial LSTMs. ArXiv:1506.03478 [Cs, Stat].
Tran, D., Hoffman, M. D., Saurous, R. A., Brevdo, E., Murphy, K., & Blei, D. M.(2017) Deep Probabilistic Programming. ArXiv:1701.03757 [Cs, Stat].
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) Pixel Recurrent Neural Networks. ArXiv:1601.06759 [Cs].
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., & Efros, A. A.(2016) Generative Visual Manipulation on the Natural Image Manifold. ArXiv:1609.03552 [Cs].