The Living Thing / Notebooks : Controllerism

“[…]Now. Where is it?”

“Where is what?”

“The time machine.”

“You’re standing in it.” said [X].

“How… does it work?” [Y] said, trying to make it sound like a casual enquiry.

“Well, it’s really terribly simple,” said [X], “it works any way you want it to. You see, the computer that runs it is a rather advanced one. In fact it is more powerful than the sum total of all the computers on this planet including — and this is the tricky part — including itself. Never really understood that bit myself, to be honest with you. But over ninety-five per cent of that power is used in simply understanding what it is you want it to do. I simply plonk my abacus down there and it understands the way I use it. I think I must have been brought up to use an abacus when I was a… well, a child, I suppose.

“[R], for instance, would probably want to use his own personal computer. If you put it down there, where the abacus is, the machine’s computer would simple take charge of it and offer you lots of nice user-friendly time-travel applications complete with pull-down menus and desk accessories if you like. Except that you point to 1066 on the screen and you’ve got the Battle of Hastings going on outside your door, er, if that’s the sort of thing you’re interested in.”

[X]’s tone of voice suggested that his own interests lay in other areas.

“It’s, er, really quite fun in its way,” he concluded. “Certainly better than television and a great deal easier to use than a video recorder. If I miss a programme I just pop back in time and watch it. I’m hopeless fiddling with all those buttons.” […]

“You have a time machine and you use it for… watching television?”

“Well, I wouldn’t use it at all if I could get the hang of the video recorder”

—Douglas Adams, spoilery details excised.

On the dark art of persuading the computer to respond intuitively to your intentions, with particular regard to music.

The input-data side of gesture recognition.

In non-musical circles they call this “physical computing”, or “natural user interfaces”, or “tangible computing”, depending upon whom they are pitching to for funding this month.

  1. I just designed an interesting digital instrument with a bunch of free control parameters.
  2. I have an interface with a different (usually much smaller) number of control parameters.
  3. there is no obvious “best”, or even immediately intuitive, mapping from one to the other

How do I plug these into each other in an intelligible, expressive way so as to perform using them?

This question is broad, vague and and comes up all the time.

Ideas I would like to explore:

Random mappings

Copula” Models

And related stuff.

Copula are an intuitive way to relate 2 or more (monotonically varying?) values by their quantiles.

The most basic one is Gaussian, where the parameter of the copula is essentially the correlation

UI design ideas

Useful software

openkinect

python

  • MIDI
    • Magenta’s MIDI interface shows how Google does it so they can be cool.
    • RtMIDI (homepage)
    • MIDO “is a library for working with MIDI messages and ports. It’s designed to be as straight forward and Pythonic as possible.”
  • OSC
    • The original, sorta-working thing: pyOSC
    • C-based and a little smoother to use: pyliblo

javascript

  • tangible.js is a resource for real-world interfaces plugging into to javascript. They intermittently publish useful reviews.

  • There are various handy GUI frameworks designed for musical control.

  • OpenSoundControl bridgets

    • osc.js is an Open Sound Control (OSC) library for JavaScript that works in both the browser and Node.js (And is still being maintained unlike many)

    • supercollider.js does this and much more.

    • OSC-JS exists, bridging websockets to OSC, but doesn’t look as maintained as osc.js. Are there others?

    • Yes. Legato.

      legato is a small node.js library written in coffeescript, but that doesn’t really matter. legato is designed to let you create beautifully simple connections between devices and software, regardless of environment.

    • chrome support Javascript MIDI natively. See also synestizer.

Lua

Reasonably comprehensive support for MIDI with decent timing in Löve2d.

Supercollider

  • mmExtensions by Martin Marier has the best-designed preset interpolation system I have seen, all so that its creator may plug a networked bath sponge into clarinet recordings.

Interesting hardware

Tablet computers

iPad, windows tablet…

For windows tablet, xotopad.

For iOs, Touchosc, Lemur…

Kinect

myo

myo is a wristband sensor that measures your muscles directly using EMG. Similar: the XTH using MMG - “which captures motion, direction and orientation sensors (integrated in a 9-DoF IMU) and muscle sound (also known as mechanomyogram or MMG)”

leapmotion

Infrared hand tracker. In my experience, not really stable enough for on-stage use, (needs better Kalman filtering) but gee it’s small and portable.

Keith McMillan fancy controllers

e.g. QUNexus, multi-dimensional midi controllers

Makeymakey

makeymakey
[Turns] everyday objects into touchpads and combine them with the internet. It’s a simple Invention Kit for Beginners and Experts doing art, engineering, and everything inbetween”

wiimote

Wiimote should be a normal HID device, but has many nasty sharp edges. So you avoid them using alternate libraries:

  • wiiuse is a library written in C that connects with several Nintendo Wii remotes. Supports motion sensing, IR tracking, nunchuk, classic controller, Balance Board, and the Guitar Hero 3 controller. Single threaded and nonblocking makes a light weight and clean API.
  • OS X mapper Darwiinremote.
  • OSX driver wjoy
  • osculator is a commerical product which does this; it’s pretty good.
  • libcwiid seems to be linux-happy? But it’s a naked C library and apparently threading-tricky. Come with a python interface

webcam

See synestizer.

Portable wireless routers

DIY

Build your own using Arduino or whatever interface.

see, e.g. mikrokontrolleur.

Graphics tablets

Wacom are the recognisable ones here, but they are crazy expensive and their drivers are shite. Presumably their competitors also have shite drivers, but they cost less. Which competitor is least worst?