Multimodal Music Stand
The Multimodal Music Stand (MMMS) is a new way to control audio synthesis and effects while playing a traditional instrument (sax, flute, violin, you name it), and offers quite a few extra possibilities for extended techniques that can be used while playing! There's already a webpage about our group project at UC Santa Barbara in which we developed the MMMS, so instead of repeating that here, I'll post the link - check it out!
Multimodal Music Stand Website
A short clip of the the MMMS in action is here, but there are better videos on the site above...
Multimodal refers to the capability of sensing input in more than one modality (audio input, video input, and sensor-based input). The audio and video input are pretty straightforward, involving a normal microphone and a webcam connected to your computer, but the sensor input is a bit more complex, hence this instructable...
We use the CREATE USB Interface, a simple circuit I developed for a class I teach at UCSB, along with the ThereminVision II kit for E-field (also known as capacitive sensing) as shown here:
CREATE USB Interface
ThereminVisionII kit
If you don't know what a Theremin is, do yourself a favor and search Youtube, you won't be dissapointed! The MMMS is basically equivalent to 2 Theremins, since it has 4 E-field sensing antennas, in addition to the audio/video input of course.
If you're interested in extending your performance techniques with your instrument by interacting with a computer, but don't want to use simple footpedals, build yourself a Multimodal Music Stand and start practicing with it!
Multimodal Music Stand Website
A short clip of the the MMMS in action is here, but there are better videos on the site above...
Multimodal refers to the capability of sensing input in more than one modality (audio input, video input, and sensor-based input). The audio and video input are pretty straightforward, involving a normal microphone and a webcam connected to your computer, but the sensor input is a bit more complex, hence this instructable...
We use the CREATE USB Interface, a simple circuit I developed for a class I teach at UCSB, along with the ThereminVision II kit for E-field (also known as capacitive sensing) as shown here:
CREATE USB Interface
ThereminVisionII kit
If you don't know what a Theremin is, do yourself a favor and search Youtube, you won't be dissapointed! The MMMS is basically equivalent to 2 Theremins, since it has 4 E-field sensing antennas, in addition to the audio/video input of course.
If you're interested in extending your performance techniques with your instrument by interacting with a computer, but don't want to use simple footpedals, build yourself a Multimodal Music Stand and start practicing with it!
About the CREATE USB Interface
The CREATE USB Interface (CUI) is a simple PIC-based programmable circuit that can either be built entirely DIY, or you can get a pre-built one from me directly for $50 (+ 5 for shipping), and you won't need to buy a PIC programmer since I put the bootloader on it for you... see the website for details, or just email me to request one:
CREATE USB Interface website
The CUI can function as many different things, and has by now been used in hundreds of interesting projects all over the world... a few examples from students at UC Santa Barbara are in this conference paper (PDF). The CUI can be reprogrammed through the USB cable using the bootloader, which is why it is so easily adaptable - simply change the firmware and it is something else.
For example, it is easy to make the CUI into a wireless sensor interface using one of the Bluetooth modules from spark fun - if anyone is interested in doing this, post a note in the comments requesting another instructable. The CUI has 13 channels of 10-bit analog inputs and 16 general purpose input/output pins. The default firmware shipped with the CUI v1.0 boards sends them all as inputs to the host computer, and works nicely with interactive art-making environments such as Max/MSP/Jitter, Pd/Gem, SuperCollider, Chuck, etc.
I used one of the CUI v1.0 boards to build the Multimodal Music Stand in this instructable. We use Max/MSP/Jitter as an environment, but by all means feel free to adapt this to Pd or your choice of software. I am a big fan of open source (the CUI itself is open source), and it helps greatly if we all stand on each other's shoulders and improve the things we're making!
CREATE USB Interface website
The CUI can function as many different things, and has by now been used in hundreds of interesting projects all over the world... a few examples from students at UC Santa Barbara are in this conference paper (PDF). The CUI can be reprogrammed through the USB cable using the bootloader, which is why it is so easily adaptable - simply change the firmware and it is something else.
For example, it is easy to make the CUI into a wireless sensor interface using one of the Bluetooth modules from spark fun - if anyone is interested in doing this, post a note in the comments requesting another instructable. The CUI has 13 channels of 10-bit analog inputs and 16 general purpose input/output pins. The default firmware shipped with the CUI v1.0 boards sends them all as inputs to the host computer, and works nicely with interactive art-making environments such as Max/MSP/Jitter, Pd/Gem, SuperCollider, Chuck, etc.
I used one of the CUI v1.0 boards to build the Multimodal Music Stand in this instructable. We use Max/MSP/Jitter as an environment, but by all means feel free to adapt this to Pd or your choice of software. I am a big fan of open source (the CUI itself is open source), and it helps greatly if we all stand on each other's shoulders and improve the things we're making!
About the ThereminVision II Kit
The ThereminVision II is available both as a kit ($50) and pre-built ($80) from the vendor site, RobotLand, and is also open source - the ThereminVision II manual (PDF) includes the circuit diagrams. For the MMMS, I bought the kit and then decided to go with the pre-built one this time (this is the second MMMS I've built).
As an aside, in case you're wondering why I don't offer a kit form of the CREATE USB Interface, it's because the CUI bootloader has to be programmed with a conventional PIC programmer before new firmware can be sent over USB - otherwise I would definitely offer a kit, though it would involve surface mount soldering...
So back to the ThereminVision II - if you bought it in kit form factor, follow the (great!) instructions in the pdf:
ThereminVision II manual
There are a lot of nice diagrams of the wiring you will need to do - the only thing it doesn't show is the connections to the CREATE USB Interface, so I will post the photos I took on the next few steps... oh, and I got the antennas for the ThereminVision (not included) from All Electronics.
As an aside, in case you're wondering why I don't offer a kit form of the CREATE USB Interface, it's because the CUI bootloader has to be programmed with a conventional PIC programmer before new firmware can be sent over USB - otherwise I would definitely offer a kit, though it would involve surface mount soldering...
So back to the ThereminVision II - if you bought it in kit form factor, follow the (great!) instructions in the pdf:
ThereminVision II manual
There are a lot of nice diagrams of the wiring you will need to do - the only thing it doesn't show is the connections to the CREATE USB Interface, so I will post the photos I took on the next few steps... oh, and I got the antennas for the ThereminVision (not included) from All Electronics.
Preparing the Music Stand
Make your Multimodal Music Stand out of a nice music stand such as the "conductor stand" shown here or use whatever you have on hand - better yet, make your own!
No matter what stand you use, it will need isolated mounts for the four antennas at the corners of the stand. Start by drilling holes in the corners - the mounting hardware I had laying around worked with a 1/4" drill bit, yours will probably be different. Try to keep the front of the surface as flush as possible to avoid interference with sheet music.
No matter what stand you use, it will need isolated mounts for the four antennas at the corners of the stand. Start by drilling holes in the corners - the mounting hardware I had laying around worked with a 1/4" drill bit, yours will probably be different. Try to keep the front of the surface as flush as possible to avoid interference with sheet music.
Connecting the ThereminVision Sensor Modules
Connecting the ThereminVision "sensor" modules (555 timer circuits) to the ThereminVision "processor" module is also discussed in more detail in the ThereminVision II manual, but here are a few pics of the process...
Connecting the ThereminVision to the CREATE USB Interface
As we saw in the last step, there are 4 wires connecting each of the sensor modules to the ThereminVision processor module. There are also 6 wires connecting the ThereminVision processor module to the CREATE USB Interface - they are Power, Ground, Sensor Output, Sensor Select A, Sensor Select B, and Sensor Stop.
Since the ThereminVision II will get it's power from the CREATE USB Interface (which in turn gets its power from USB), I decided to connect the power for each of the ThereminVision sensor modules directly to the CREATE USB Interface - this makes it a bit less of a bird's nest on the wiring since there are already a lot of wires going onto the TherminVision processor board.
Connect "+" on the ThereminVision to "5V" on the CUI
Connect "-" on the ThereminVision to "GND" on the CUI
Connect "4" (the divide by 4 output) on the ThereminVision to "D7" on the CUI
Connect "SEL B" on the ThereminVision to "D5" on the CUI
connect "SEL A" on the ThereminVision to "D4" on the CUI
Connect "S" on the ThereminVision to "D6" on the CUI
Since the ThereminVision II will get it's power from the CREATE USB Interface (which in turn gets its power from USB), I decided to connect the power for each of the ThereminVision sensor modules directly to the CREATE USB Interface - this makes it a bit less of a bird's nest on the wiring since there are already a lot of wires going onto the TherminVision processor board.
Connect "+" on the ThereminVision to "5V" on the CUI
Connect "-" on the ThereminVision to "GND" on the CUI
Connect "4" (the divide by 4 output) on the ThereminVision to "D7" on the CUI
Connect "SEL B" on the ThereminVision to "D5" on the CUI
connect "SEL A" on the ThereminVision to "D4" on the CUI
Connect "S" on the ThereminVision to "D6" on the CUI
Mounting Everything to the Stand
So the electronics is done, now just mounting it to the stand - there are many methods for doing this, and you may want to put your electronics in a protective box, etc....
Here are some photos of some of the simplest mounting methods.
Here are some photos of some of the simplest mounting methods.
Bootloading the Firmware for the ThereminVision Into the CUI
The ThereminVision sends timing signals for its sensors instead of analog outputs (or any of the standard digital protocols such as I2C or SPI). So some custom firmware to measure the timing of these signals was needed...
The first version of the Multimodal Music Stand firmware is in the zip file below, both the source code and the compiled hex file are included (along with a test patch for Max/MSP/Jitter):
CUI-ThereminVision firmware
The screenshot below shows the OS X program Boot Down by Craig Schimmel - a Macintosh bootloading utility for the CUI. If you are on windows, the easiest thing is to use Microchip's own tool, PDFSUSB.exe that is downloadable from their website, or included in this example code for the CREATE USB Interface.
To put the CUI in "bootloader mode", hold down the program button and hit the reset button (or just plug in the USB cable while holding the program button down). The CUI's status LED will blink continuously indicating that it is in bootloader mode. Now you can use Boot Down to send the CUIEfieldBootDown.hex file to the CREATE USB Interface.
If you ever want to change the firmware for the CREATE USB Interface back to the original, it is here, and there are other examples of firmware for the CUI here as well.
The first version of the Multimodal Music Stand firmware is in the zip file below, both the source code and the compiled hex file are included (along with a test patch for Max/MSP/Jitter):
CUI-ThereminVision firmware
The screenshot below shows the OS X program Boot Down by Craig Schimmel - a Macintosh bootloading utility for the CUI. If you are on windows, the easiest thing is to use Microchip's own tool, PDFSUSB.exe that is downloadable from their website, or included in this example code for the CREATE USB Interface.
To put the CUI in "bootloader mode", hold down the program button and hit the reset button (or just plug in the USB cable while holding the program button down). The CUI's status LED will blink continuously indicating that it is in bootloader mode. Now you can use Boot Down to send the CUIEfieldBootDown.hex file to the CREATE USB Interface.
If you ever want to change the firmware for the CREATE USB Interface back to the original, it is here, and there are other examples of firmware for the CUI here as well.
Testing the Multimodal Music Stand in Max/MSP/Jitter
Open the example patch in Max/MSP/Jitter (a 30-day demo is downloadable from www.cycling74.com, and a "runtime" version also available that doesn't allow saving patches), or follow the conventions in the Max patch and make your own patch in PureData (Pd)...
Jitter is used to display the 3D position of gestural inputs, where the z-axis (towards and away from the stand) is mapped to the overall intensity of all 4 antennas - not very scientific, but it works! The data from the individual sensors is also sent out as OSC to 127.0.0.1 (localhost) for use by other applications if needed.
A useful object in Max/MSP/Jitter for filtering spikes out of the sensor data (caused by electrical interference) is Tristan Jehan's "smoother", which is downloadable here:
smoother max object
Tristan's site also has a bunch of MSP externals for Max that we use for the audio analysis portion of the Multimodal Music Stand's input - they include pitch~, loudness~, brightness~, noisiness~, and the all-in-one analyzer~ object.
Once you have data coming into Max, you'll need to adjust the sensitivity of the ThereminVision sensors individually with a small srewdriver. Do this by extending the antennas all the way, then turning the blue trimpot counterclockwise until it goes into "reverse sensing" mode - then turn it back clockwise until you're back in normal sensing range. Small adjustments (tuning the senstivity, just like tuning a guitar) can be made by changing the length of the antennas, so you won't need to carry the tiny screwdriver around to gigs with you.
Jitter is used to display the 3D position of gestural inputs, where the z-axis (towards and away from the stand) is mapped to the overall intensity of all 4 antennas - not very scientific, but it works! The data from the individual sensors is also sent out as OSC to 127.0.0.1 (localhost) for use by other applications if needed.
A useful object in Max/MSP/Jitter for filtering spikes out of the sensor data (caused by electrical interference) is Tristan Jehan's "smoother", which is downloadable here:
smoother max object
Tristan's site also has a bunch of MSP externals for Max that we use for the audio analysis portion of the Multimodal Music Stand's input - they include pitch~, loudness~, brightness~, noisiness~, and the all-in-one analyzer~ object.
Once you have data coming into Max, you'll need to adjust the sensitivity of the ThereminVision sensors individually with a small srewdriver. Do this by extending the antennas all the way, then turning the blue trimpot counterclockwise until it goes into "reverse sensing" mode - then turn it back clockwise until you're back in normal sensing range. Small adjustments (tuning the senstivity, just like tuning a guitar) can be made by changing the length of the antennas, so you won't need to carry the tiny screwdriver around to gigs with you.
Using the Multimodal Music Stand in Performance! (and Extending It)
Don't forget, the new performance techniques that the Multimodal Music Stand enables need practice, and development of new mappings and ideas for what you want to control with it!
Also, if you want to extend your MMMS to incorporate other kinds of sensors, it will be possible to use the 13 analog inputs on the CUI in addition to the ThereminVision sensors in future versions of the MMMS firmware, so with some simple sensors (sliders/knobs/footpedals) or other sensor types (IR/ultrasound/etc), these can be used along with audio input and video to the computer. Currently the firmware uses a "serial-over-USB" protocol but eventually the plan is to switch to the "HID" (Human Input Device) protocol normally used by the CREATE USB Interface. Please feel free to contribute to the effort...
There is a similar open source project for E-field sensing (also known as capacitive sensing) that uses the same circuit topology (based on the ThereminVision design) called CapToolKit - it uses the "little brother" of the PIC18F4550, the PIC18F2550 which doesn't have as many analog inputs (10 instead of 13). The firmware for the CapToolKit will be slightly different since they don't use the ThereminVision processor board.
We use a simple firewire camera (webcam style) mounted on the top of the music stand for optical recognition of physical gestures and gaze detection - If you are interested in the computer vision or audio analysis and synthesis aspects of our group's research, please see the papers at the Multimodal Music Stand website.
And please do post your experiences if you build a Multimodal Music Stand... We now have two of them at UCSB as shown in the photo below - it would be great to have more compositions, performances, and musicians using them out there! Dr. JoAnn Kuchera-Morin wrote the first composition involving the MMMS, which was performed by the flutist Jill Felber at the Eastman School of Music in March 2007.
Also, if you want to extend your MMMS to incorporate other kinds of sensors, it will be possible to use the 13 analog inputs on the CUI in addition to the ThereminVision sensors in future versions of the MMMS firmware, so with some simple sensors (sliders/knobs/footpedals) or other sensor types (IR/ultrasound/etc), these can be used along with audio input and video to the computer. Currently the firmware uses a "serial-over-USB" protocol but eventually the plan is to switch to the "HID" (Human Input Device) protocol normally used by the CREATE USB Interface. Please feel free to contribute to the effort...
There is a similar open source project for E-field sensing (also known as capacitive sensing) that uses the same circuit topology (based on the ThereminVision design) called CapToolKit - it uses the "little brother" of the PIC18F4550, the PIC18F2550 which doesn't have as many analog inputs (10 instead of 13). The firmware for the CapToolKit will be slightly different since they don't use the ThereminVision processor board.
We use a simple firewire camera (webcam style) mounted on the top of the music stand for optical recognition of physical gestures and gaze detection - If you are interested in the computer vision or audio analysis and synthesis aspects of our group's research, please see the papers at the Multimodal Music Stand website.
And please do post your experiences if you build a Multimodal Music Stand... We now have two of them at UCSB as shown in the photo below - it would be great to have more compositions, performances, and musicians using them out there! Dr. JoAnn Kuchera-Morin wrote the first composition involving the MMMS, which was performed by the flutist Jill Felber at the Eastman School of Music in March 2007.