I want to recognise gestures made with generic interface devices for artistic purposes, in realtime. Is that so much to ask?
Related: synestizer, time warping, functional data analysis
- Gesture variation following has particular algorithms optimised for realtime music an video control using AFAICT particle filter. This is a different approach to the other ones, which use off-the-shelf algorithms for the purpose, which leads to some difficulties. (source is c++, puredata and maxmsp interfaces available)
The Gesture Recognition Toolkit other software for gesture recognition; lower level than wekinator (default API is raw C++), more powerful algorithsm, although a less beguiling demo video.
Now also includes a GUI and puredata opensoundcontrol interfaces in addition to the original C++ API.
- Interesting application: generic myo control
- Eyesweb: An inscrutably under-explained GUI(?) for integrating UI stuff somehow or other.
- Wekinator: Software for using machine learning to build real-time interactive systems. (Which is to say, a workflow optimised for ad-hoc, slippery, artsy applications of cold, hard, calculating machine learning techniques.)
- Beautifully simple “graffiti” letter recogniser (NN-search on normalised characters, neat hack. Why you should always start from the simplest thing.) (via Chr15m)
- how teh kinext recognises (spoiler: random forests)
BTW, you can also roll your own with any machine learning library; It’s not clear how much you need all the fancy time-warping tricks.
Likely bottlenecks are constructing a training data set and getting the damn thing to work in realtime. I should make some notes on that theme.
Apropos that Museplayer can record opensoundcontrol data.
- Caramiaux, B., Montecchio, N., Tanaka, A., & Bevilacqua, F. (2014) Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Trans. Interact. Intell. Syst., 4(4), 18:1–18:34. DOI.
- Chen, F.-S., Fu, C.-M., & Huang, C.-L. (2003) Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing, 21(8), 745–758. DOI.
- Criminisi, A., Shotton, J., & Konukoglu, E. (2011) Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning (No. MSR-TR-2011-114). . Microsoft Research
- Criminisi, Antonio, Shotton, J., & Konukoglu, E. (2012) Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Foundations and Trends® in Computer Graphics and Vision, 7(2–3). DOI.
- Fiebrink, R., & Cook, P. R.(2010) The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010). Utrecht.
- Fiebrink, R., Cook, P. R., & Trueman, D. (2011) Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 147–156). New York, NY, USA: ACM DOI.
- Fiebrink, R., Trueman, D., & Cook, P. R.(2009) A metainstrument for interactive, on-the-fly machine learning. In Proceefdings of NIME (Vol. 2, p. 3).
- Françoise, J., Schnell, N., Borghesi, R., & Bevilacqua, F. (2014) Probabilistic Models for Designing Motion and Sound Relationships. In Proceedings of the 2014 International Conference on New Interfaces for Musical Expression (pp. 287–292). London, UK, United Kingdom
- Gillian, N., Knapp, B., & O’Modhrain, S. (2011a) Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping.
- Gillian, N., Knapp, R., & O’Modhrain, S. (2011b) A machine learning toolbox for musician computer interaction. NIME11.
- Hong, P., Turk, M., & Huang, T. S.(2000) Gesture modeling and recognition using finite state machines. In Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings (pp. 410–415). DOI.
- Kratz, S., & Rohs, M. (2010) A $3 Gesture Recognizer: Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (pp. 341–344). New York, NY, USA: ACM DOI.
- Lee, H.-K., & Kim, J. H.(1999) An HMM-based threshold model approach for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 961–973. DOI.
- Marković, D., Valčić, B., & Malešević, N. (2016) Body movement to sound interface with vector autoregressive hierarchical hidden Markov models. arXiv:1610.08450 [Cs, Stat].
- Mitra, S., & Acharya, T. (2007) Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3), 311–324. DOI.
- Murakami, K., & Taguchi, H. (1991) Gesture recognition using recurrent neural networks. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 237–242). ACM
- Rocchesso, D., Lemaitre, G., Susini, P., Ternström, S., & Boussard, P. (2015) Sketching Sound with Voice and Gesture. Interactions, 22(1), 38–41. DOI.
- Schacher, J. C.(2015) Gestural Electronic Music using Machine Learning as Generative Device. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME’15,. Baton Rouge, USA: Louisiana State University
- Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008) Gesture Recognition with a Wii Controller. In Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction (pp. 11–14). New York, NY, USA: ACM DOI.
- Wang, S. B., Quattoni, A., Morency, L., Demirdjian, D., & Darrell, T. (2006) Hidden Conditional Random Fields for Gesture Recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 1521–1527). IEEE DOI.
- Williamson, J., & Murray-Smith, R. (2002) Audio feedback for gesture recognition.
- Wilson, A. D., & Bobick, A. F.(1999) Parametric hidden Markov models for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(9), 884–900. DOI.
- Wu, Y., & Huang, T. S.(1999) Vision-Based Gesture Recognition: A Review. In A. Braffort, R. Gherbi, S. Gibet, D. Teil, & J. Richardson (Eds.), Gesture-Based Communication in Human-Computer Interaction (pp. 103–115). Springer Berlin Heidelberg
- Yang, M.-H., Ahuja, N., & Tabb, M. (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1061–1074. DOI.
- Yoon, H.-S., Soh, J., Bae, Y. J., & Seung Yang, H. (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognition, 34(7), 1491–1501. DOI.