The Living Thing / Notebooks :

Gesture recognition

I want to recognise gestures made with generic interface devices for artistic purposes, in realtime. Is that so much to ask?

Related: synestizer, time warping, functional data analysis

To Use

BTW, you can also roll your own with any machine learning library; It’s not clear how much you need all the fancy time-warping tricks.

Likely bottlenecks are constructing a training data set and getting the damn thing to work in realtime. I should make some notes on that theme.

Apropos that Museplayer can record opensoundcontrol data.


Caramiaux, B., Montecchio, N., Tanaka, A., & Bevilacqua, F. (2014) Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Trans. Interact. Intell. Syst., 4(4), 18:1–18:34. DOI.
Chen, F.-S., Fu, C.-M., & Huang, C.-L. (2003) Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing, 21(8), 745–758. DOI.
Criminisi, A., Shotton, J., & Konukoglu, E. (2011) Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning (No. MSR-TR-2011-114). . Microsoft Research
Criminisi, Antonio, Shotton, J., & Konukoglu, E. (2012) Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Foundations and Trends® in Computer Graphics and Vision, 7(2–3). DOI.
Fiebrink, R., & Cook, P. R.(2010) The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010). Utrecht.
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 147–156). New York, NY, USA: ACM DOI.
Fiebrink, R., Trueman, D., & Cook, P. R.(2009) A metainstrument for interactive, on-the-fly machine learning. In Proceefdings of NIME (Vol. 2, p. 3).
Françoise, J., Schnell, N., Borghesi, R., & Bevilacqua, F. (2014) Probabilistic Models for Designing Motion and Sound Relationships. In Proceedings of the 2014 International Conference on New Interfaces for Musical Expression (pp. 287–292). London, UK, United Kingdom
Gillian, N., Knapp, B., & O’Modhrain, S. (2011a) Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping.
Gillian, N., Knapp, R., & O’Modhrain, S. (2011b) A machine learning toolbox for musician computer interaction. NIME11.
Hong, P., Turk, M., & Huang, T. S.(2000) Gesture modeling and recognition using finite state machines. In Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings (pp. 410–415). DOI.
Kratz, S., & Rohs, M. (2010) A $3 Gesture Recognizer: Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (pp. 341–344). New York, NY, USA: ACM DOI.
Lee, H.-K., & Kim, J. H.(1999) An HMM-based threshold model approach for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 961–973. DOI.
Marković, D., Valčić, B., & Malešević, N. (2016) Body movement to sound interface with vector autoregressive hierarchical hidden Markov models. arXiv:1610.08450 [Cs, Stat].
Mitra, S., & Acharya, T. (2007) Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3), 311–324. DOI.
Murakami, K., & Taguchi, H. (1991) Gesture recognition using recurrent neural networks. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 237–242). ACM
Rocchesso, D., Lemaitre, G., Susini, P., Ternström, S., & Boussard, P. (2015) Sketching Sound with Voice and Gesture. Interactions, 22(1), 38–41. DOI.
Schacher, J. C.(2015) Gestural Electronic Music using Machine Learning as Generative Device. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME’15,. Baton Rouge, USA: Louisiana State University
Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008) Gesture Recognition with a Wii Controller. In Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction (pp. 11–14). New York, NY, USA: ACM DOI.
Wang, S. B., Quattoni, A., Morency, L., Demirdjian, D., & Darrell, T. (2006) Hidden Conditional Random Fields for Gesture Recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 1521–1527). IEEE DOI.
Williamson, J., & Murray-Smith, R. (2002) Audio feedback for gesture recognition.
Wilson, A. D., & Bobick, A. F.(1999) Parametric hidden Markov models for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(9), 884–900. DOI.
Wu, Y., & Huang, T. S.(1999) Vision-Based Gesture Recognition: A Review. In A. Braffort, R. Gherbi, S. Gibet, D. Teil, & J. Richardson (Eds.), Gesture-Based Communication in Human-Computer Interaction (pp. 103–115). Springer Berlin Heidelberg
Yang, M.-H., Ahuja, N., & Tabb, M. (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1061–1074. DOI.
Yoon, H.-S., Soh, J., Bae, Y. J., & Seung Yang, H. (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognition, 34(7), 1491–1501. DOI.