The Living Thing / Notebooks : Gesture recognition

I want to recognise gestures made with generic interface devices for artistic purposes, in realtime. Is that so much to ask?

Related: synestizer, time warping, functional data analysis

To Use

BTW, you can also roll your own with any machine learning library; It’s not clear how much you need all the fancy time-warping tricks.

Likely bottlenecks are constructing a training data set and getting the damn thing to work in realtime. I should make some notes on that theme.

Apropos that Museplayer can record opensoundcontrol data.

Refs

CMTB14
Caramiaux, B., Montecchio, N., Tanaka, A., & Bevilacqua, F. (2014) Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Trans. Interact. Intell. Syst., 4(4), 18:1–18:34. DOI.
ChFH03
Chen, F.-S., Fu, C.-M., & Huang, C.-L. (2003) Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing, 21(8), 745–758. DOI.
CrSK11
Criminisi, A., Shotton, J., & Konukoglu, E. (2011) Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning (No. MSR-TR-2011-114). . Microsoft Research
CrSK12
Criminisi, Antonio, Shotton, J., & Konukoglu, E. (2012) Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Foundations and Trends® in Computer Graphics and Vision, 7(2–3). DOI.
FiCo10
Fiebrink, R., & Cook, P. R.(2010) The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010). Utrecht.
FiCT11
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 147–156). New York, NY, USA: ACM DOI.
FiTC09
Fiebrink, R., Trueman, D., & Cook, P. R.(2009) A metainstrument for interactive, on-the-fly machine learning. In Proceefdings of NIME (Vol. 2, p. 3).
FSBB14
Françoise, J., Schnell, N., Borghesi, R., & Bevilacqua, F. (2014) Probabilistic Models for Designing Motion and Sound Relationships. In Proceedings of the 2014 International Conference on New Interfaces for Musical Expression (pp. 287–292). London, UK, United Kingdom
GiKO11a
Gillian, N., Knapp, B., & O’Modhrain, S. (2011a) Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping.
GiKO11b
Gillian, N., Knapp, R., & O’Modhrain, S. (2011b) A machine learning toolbox for musician computer interaction. NIME11.
HoTH00
Hong, P., Turk, M., & Huang, T. S.(2000) Gesture modeling and recognition using finite state machines. In Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings (pp. 410–415). DOI.
KrRo10
Kratz, S., & Rohs, M. (2010) A $3 Gesture Recognizer: Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (pp. 341–344). New York, NY, USA: ACM DOI.
LeKi99
Lee, H.-K., & Kim, J. H.(1999) An HMM-based threshold model approach for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 961–973. DOI.
MaVM16
Marković, D., Valčić, B., & Malešević, N. (2016) Body movement to sound interface with vector autoregressive hierarchical hidden Markov models. arXiv:1610.08450 [Cs, Stat].
MiAc07
Mitra, S., & Acharya, T. (2007) Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3), 311–324. DOI.
MuTa91
Murakami, K., & Taguchi, H. (1991) Gesture recognition using recurrent neural networks. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 237–242). ACM
RLST15
Rocchesso, D., Lemaitre, G., Susini, P., Ternström, S., & Boussard, P. (2015) Sketching Sound with Voice and Gesture. Interactions, 22(1), 38–41. DOI.
Scha15
Schacher, J. C.(2015) Gestural Electronic Music using Machine Learning as Generative Device. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME’15,. Baton Rouge, USA: Louisiana State University
SPHB08
Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008) Gesture Recognition with a Wii Controller. In Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction (pp. 11–14). New York, NY, USA: ACM DOI.
WQMD06
Wang, S. B., Quattoni, A., Morency, L., Demirdjian, D., & Darrell, T. (2006) Hidden Conditional Random Fields for Gesture Recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 1521–1527). IEEE DOI.
WiMu02
Williamson, J., & Murray-Smith, R. (2002) Audio feedback for gesture recognition.
WiBo99
Wilson, A. D., & Bobick, A. F.(1999) Parametric hidden Markov models for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(9), 884–900. DOI.
WuHu99
Wu, Y., & Huang, T. S.(1999) Vision-Based Gesture Recognition: A Review. In A. Braffort, R. Gherbi, S. Gibet, D. Teil, & J. Richardson (Eds.), Gesture-Based Communication in Human-Computer Interaction (pp. 103–115). Springer Berlin Heidelberg
YaAT02
Yang, M.-H., Ahuja, N., & Tabb, M. (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1061–1074. DOI.
YSBS01
Yoon, H.-S., Soh, J., Bae, Y. J., & Seung Yang, H. (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognition, 34(7), 1491–1501. DOI.