I want to recognise gestures made with generic interface devices for artistic purposes, in realtime. Is that so much to ask?
Related: synestizer, time warping, functional data analysis
Gesture variation following has particular algorithms optimised for realtime music an video control using AFAICT particle filter. This is a different approach to the other ones, which use off-the-shelf algorithms for the purpose, which leads to some difficulties. (source is c++, puredata and maxmsp interfaces available)
GRT: The Gesture Recognition Toolkit other software for gesture recognition; lower level than wekinator (default API is raw C++), more powerful algorithsm, although a less beguiling demo video. Now also includes a GUI and puredata opensoundcontrol interfaces in addition to the original C++ API.
- Interesting application: generic myo control
Eyesweb: An inscrutably under-explained GUI(?) for integrating UI stuff somehow or other.
Wekinator: Software for using machine learning to build real-time interactive systems. (Which is to say, a workflow optimised for ad-hoc, slippery, artsy applications of cold, hard, calculating machine learning techniques.)
BTW, you can also roll your own with any machine learning library; It’s not clear how much you need all the fancy time-warping tricks.
Likely bottlenecks are constructing a training data set and getting the damn thing to work in realtime. I should make some notes on that theme.
Apropos that Museplayer can record opensoundcontrol data.
skatvg got a grant to make people’s gesturing and whooshing and beatboxing produce real soundtracks.
- KrRo10: (2010) A $3 Gesture Recognizer: Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In Proceedings of the 15th International Conference on Intelligent User Interfaces (pp. 341–344). New York, NY, USA: ACM DOI
- GiKO11a: (2011a) A machine learning toolbox for musician computer interaction. NIME11.
- FiTC09: (2009) A metainstrument for interactive, on-the-fly machine learning. In Proceefdings of NIME (Vol. 2, p. 3).
- CMTB14: (2014) Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Trans. Interact. Intell. Syst., 4(4), 18:1–18:34. DOI
- LeKi99: (1999) An HMM-based threshold model approach for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 961–973. DOI
- WiMu02: (2002) Audio feedback for gesture recognition.
- MaVM16: (2016) Body movement to sound interface with vector autoregressive hierarchical hidden Markov models. ArXiv:1610.08450 [Cs, Stat].
- CrSK12: (2012) Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Foundations and Trends® in Computer Graphics and Vision, 7(2–3). DOI
- CrSK11: (2011) Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning (No. MSR-TR-2011-114). Microsoft Research
- YaAT02: (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1061–1074. DOI
- Scha15: (2015) Gestural Electronic Music using Machine Learning as Generative Device. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME’15,. Baton Rouge, USA: Louisiana State University
- HoTH00: (2000) Gesture modeling and recognition using finite state machines. In Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings (pp. 410–415). DOI
- MiAc07: (2007) Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3), 311–324. DOI
- MuTa91: (1991) Gesture recognition using recurrent neural networks. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 237–242). ACM
- SPHB08: (2008) Gesture Recognition with a Wii Controller. In Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction (pp. 11–14). New York, NY, USA: ACM DOI
- ChFH03: (2003) Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing, 21(8), 745–758. DOI
- YSBS01: (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognition, 34(7), 1491–1501. DOI
- WQMD06: (2006) Hidden Conditional Random Fields for Gesture Recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 1521–1527). IEEE DOI
- KiPR00: (10000) How the Chinese Government Fabricates Social Media Posts for Strategic Distraction, not Engaged Argument. American Political Science Review.
- FiCT11: (2011) Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 147–156). New York, NY, USA: ACM DOI
- Pain02: (2002) Interactivity, where to from here? Organised Sound, 7(3), 295–304. DOI
- HuWa02: (2002) Mapping performer parameters to synthesis engines. Organised Sound, 7(2), 97–108. DOI
- Wrig05: (2005) Open Sound Control: an enabling technology for musical networking. Organised Sound, 10(3), 193–200. DOI
- WiBo99: (1999) Parametric hidden Markov models for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(9), 884–900. DOI
- FSBB14: (2014) Probabilistic Models for Designing Motion and Sound Relationships. In Proceedings of the 2014 International Conference on New Interfaces for Musical Expression (pp. 287–292). London, UK, United Kingdom
- GiKO11b: (2011b) Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping.
- RLST15: (2015) Sketching Sound with Voice and Gesture. Interactions, 22(1), 38–41. DOI
- ACKV02: (2002) Strategies of mapping between gesture data and synthesis model parameters using perceptual spaces. Organised Sound, 7(2), 127–144. DOI
- CDPS17: (2017) The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. 26th WWW.
- FiCo10: (2010) The Wekinator: a system for real-time, interactive machine learning in music. In Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010). Utrecht.
- WuHu99: (1999) Vision-Based Gesture Recognition: A Review. In Gesture-Based Communication in Human-Computer Interaction (pp. 103–115). Springer Berlin Heidelberg