Puppeteer Motion-Capture Costume

Exploring the use of affordable, off-the-shelf materials and basic techniques to create wireless motion capture wearables.

The idea behind Puppeteer is to create accessible wearable technology solutions for motion-capture, aiming to create as much of the technology from scratch, collecting and sharing this knowledge through DIY instructions. The name Puppeteer comes from the concept of being able to puppeteer or control. In this case, the motion of the body wearing the costume controls whatever data is relevant to the performance or project.
The fabrication of the suit is a handmade procedure, which is not intended for mass-production, but rather for small projects lead by individuals with enthusiasm for making things themselves, sewing, gluing, soldering, programming and bug fixing.

The Puppeteer project is continuously being developed by Mika Satomi and Hannah Perner-Wilson http://www.KOBAKANT.at, and as such is constantly being expanded and refined. We welcome any feedback and input from interested individuals, groups and companies.

This Instructable goes into as much detail as we think makes sense. When following the steps you must be prepared to solve some of the problems such as designing a pattern, deciding on sensor placement, planning your circuit by yourself.
This Instructable explains the techniques we applied to bring things together and create a motion-capture wearable. The aim here is not to recreate our Puppeteer costume but to make your own. And hopefully find solutions to existing problems!

PREREQUISITES
- Sewing
- Soldering
- Pattern making
- Basic understanding of the Microcontrollers and the Arduinio, multiplexers, circuits and components
- Basic understanding of code and communicating with serial devices

MATERIALS
- Conductive thread from http://www.sparkfun.com
also see http://cnmat.berkeley.edu/resource/conductive_thread
- Neoprene from http://www.sedochemicals.com
- Stretch conductive fabric from http://www.lessemf.com
also see http://cnmat.berkeley.edu/resource/stretch_conductive_fabric
- Fusible interfacing from local fabric store or
also see http://www.shoppellon.com
- Velostat by 3M from www.lessemf.com
also see http://cnmat.berkeley.edu/resource/velostat_resistive_plastic
- Machine poppers/snaps from local fabric store
- Regular sewing thread from local fabric store
- Amounts of stretch fabrics you will need to make the costume from
- Perfboard with copper line pattern from http://www.allelectronics.com
- Wire
- Shrink tubing
- Arduino LilyPad from http://www.sparkfun.com
- Two XBee modules - one for the suit and one communicating with it
- Two 3.7V Lipo batteries and chargers

TOOLS:
- Pens and lots of paper
- Fabric pen that disappears over time
- Ruler, soft measuring tape
- Fabric scissors and paper scissors
- Sewing machine that can do the stretchy stitch
- Iron
- Sewing needles
- Popper/snap machine - handheld or hammer and simple version
- Pliers for undoing poppers
- Soldering iron and solder
- Helping hands
- Wire clippers
- Wire strippers

The principle behind the Puppeteer motion capture costume is to explore the use of fabrication methods that use very basic materials and tools and yet still produce useful solutions for motion capture. We have so far focused on capturing motion of the human body, but the techniques we have developed could also be applied to stuffed toys, animals and other objects.

The puppeteer suit has been developed as an interface for various projects, adhering to requirements that were not purely motion capture oriented. The main focus throughout these projects has not been to collect precise data on the position of every limb and joint of the wearer, but rather to collect motion data such as joint angle and velocity from specific areas of the body. This data, as well as being translated into various outcome, can also be intentionally manipulated by the wearer, allowing them to interact with the system in a meaningful way.

So far there have been three completed versions of Puppeteer motion-capture costumes that have been used in the following performances:
1) Ein Kleines Puppenspiel, 2) Language Game at LEMUR and 3) Perfect Human
More information about all these performances can be found on http://www.KOBAKANT.at and in the next step.

1) Ein Kleines Puppenspiel
In this performance the performer, Ivana Kalc http://myspace.com/ivanakalc wearing the costume was moving her body in order to puppeteer a representation of herself in a digital world. This performance was called Ein Kleines Puppenspiel http://puppenspiel.moviesandbox.net/ and was conceived by Friedrich Kirschner, who conceived and created the narrative, using a computer game engine Unreal Tournament, in which characters, cameras and events are all triggered by human actors acting in the real world through various devices, playing out the story, a real time puppet play.
After producing the first working Puppeteer costume for this project, we decided to continue developing the Puppeteer costume. We started to explore new ways of exploiting, manipulating and designing conductive and resistive textiles to sense bodily motion, as well as creating soft and stretchy conductive traces. We wanted to be able to produce comfortable and washable costumes that, unless intended, did not limit the wearer's natural scope of motion.


2) Language Game at LEMUR
The next Puppeteer project was developed during a one-month residency at LEMUR http://lemurbots.org/ in Brooklyn, NY. LEMUR has a set up of musical robots that can be triggered by Midi-signal. Our plan was to have a performer, wearing a motion capture costume, trigger various musical robots with her movements, creating a sound-scape to which in turn would influence her movements. We spent 4 intense weeks refining this concept, as well as designing and fabricating the next Puppeteer suit. We worked with a local ballroom dancer, Micaela Schedlbauer, who was interested in the project. Michaela had a very set dance routine and it was interesting to work with her, though unfortunately the time-frame did not really leave space for enough practice to break the routine in a way that gave her enough confidence to listen to the system, and to react to what was going on.


3) Perfect Human Performance
The third Puppeteer costume was created for a performance inspired by a 1967 short film by Joergen Leth titled The Perfect Human and by Lars von Trier's 2003 The Five Obstructions. The performance took the text from the original film and introduced the obstructions of performance and non-linear narration. The original text was cut up into blocks and the performer was presented with a game like situation in which certain words and segments of sentences were mapped to the movement of certain body parts. The performer had to first figure out what movements triggered what fragments before she could try to speak the full sentence. Once a sentence was intentionally spoken correctly the performer would move on to the next level, the next sentence and a new mapping.
This Puppeteer project was first developed for the Interface Cultures exhibition during the 2008 Ars Electronica in Linz, Austria, the performer was Ivana Kalc. Plans for future performances of the Perfect Human Performance intend collaboration with local performers in various locations. The only setback is that the performers must fit the existing Perfect Human Puppeteer suit, though slight alterations are possible.

All three Puppeteer suits to date are based on the principle that pressure sensors also react to bend. This is true since pressure is exerted thorough the fact that materials pressure against each other when bent because the outer layer must stretch and presses against the inner layer, which has excess material that needs to go somewhere.

Shortly prior to the first Puppeteer project we had developed the technique of making pressure sensors sandwiched between two layers of neoprene. The reason we chose to use neoprene was because we had been using it to make fabric buttons for another project http://www.massage-me.at Neoprene has some nice qualities for making pressure sensors since it creates a natural force feedback as well as spreading the pressure evenly.

Fabric Bend Sensor
The first fabric bend sensors we fabricated from two layers of 1.5mm neoprene, which has polyester jersey adhered to each side. The fact that fabric was mounted on either side of the actually neoprene made it possible to fuse a piece of conductive fabric to either piece of neoprene using iron-on interfacing, a heat glue common in quilting and sewing. In between the two pieces of conductive fabric we sandwiched a piece of Velostat, which is basically the black plastic that sensitive electronic components or circuits are packaged in to protect them from electrical charges or statically charging themselves. Velostat is carbon-impregnated polyolefin and is a bad conductor but also has the quality to reduce its resistance under pressure.
All these layers are kept in place by lightly stitching around the edges of the neoprene.
In the first Puppeteer suit we had twelve such sensors places around the body. Although the sensors are stable, they do such a lot of power, even when they are not being bent, since there is enough initial pressure on the sensor so that there is a constant flow of electricity that increases even more when bent.

Conductive Thread Bend Sensor
There is an Instructable that covers this Bend sensor >> https://www.instructables.com/id/Fabric_bend_sensor/

One change that was made in the next Puppeteer suit was to reduce the conductive surfaces on either side of the Velostat. So instead of fusing conductive fabric, we stitches a few diagonal stitches using conductive thread on either side of the neoprene. This reduced the initial conductivity a lot so that some of the sensors that were not mounted to tightly against the body, made no electrical connection unless pressured/bent.
Also, circuit wise instead of power all sensors and measuring the changes in power passing through them we used a multiplexer to individually power and measure each sensor individually in super fast succession, so seemingly seamlessly, all at once. This, along with the conductive thread alternative lowered the energy consumption to a good level.

I am also selling these handmade fabric bend sensors via Etsy. Although it is much cheaper to make your own, purchasing one will help me support my prototyping and development costs >>
http://www.etsy.com/shop.php?user_id=5178109

Even though we have been exploring new materials, combinations and designs that produce pressure, bend and stretch sensitive sensors we have not yet implemented these results into a full Puppeteer suit. The following experiments offer some interesting starting points for further development, though we have failed to implement them in a full suit because of various reasons such as not being able to reproduce the results with enough consistency. Even though none of these experiments have yet been implemented in a Puppeteer suit, they are still part of the Puppeteer project and they will hopefully make their way into future versions.

Carbon Paint
One technique that produces some lovely results is the application of carbon paint to fabric. Depending on the thickness of the paint and the underlying fabric, one achieves different results. If the paint starts to crack too much, then when stretched the conductivity sinks, since there is simply just less physical connection. But if things come out just right and there is not too much cracking then the conductivity increases when stretched and we believe this is due to the fact that the carbon particles are being compressed though the physicality of the stretching and thus improving the conductive connection.

Conductive Textiles
The Berkley based company Eeonyx http://www.eeonyx.com/ produces some very interesting conductive textiles. They have the capability to coat almost any material/textile in a secret polymer solution, which can be regulated to produce any desired level of electrical conductivity/resistance. When they coat stretchy knits and materials such as Lycra and nylon, they practically produce stretch sensors. This is something we have just started to explore

Conductive Rubbers
There are lots of different conductive rubbers on the market and we have tested a few of them:
- Conductive rubber bands sold by All-Spec http://www.all-spec.com/1/viewitem/1862C/ALLSPEC/prodinfo/i=rss, also see: http://cnmat.berkeley.edu/resource/conductive_black_rubber_band
- Zoflex conductive rubber http://www.rfmicrolink.com/Conductive_rubber.html, also see: http://cnmat.berkeley.edu/resource/zoflex_conductive_rubber
- Conductive rubber ropes from Images Scientific Instruments http://www.imagesco.com/sensors/stretch-sensor.html

The problem we have had with using these rubbers as stretch sensors is that they are either very tough and take a lot of pull to really stretch and get a nice change in resistance, and as such would be hard to integrate in a motion capture suit, unless we make it super tight (or come up with other solutions). The other problem we encounter is that even though the change in resistance is measurable it bounces around a bit. It is often slow to react and then bounces to a peak before dropping down to a certain point, even though the rubber has only been stretches to a certain degree and held in the same position. These rubbers all react very nicely to pressure, when sandwiched between two conductive layers. They make great pressure sensors, replacing the layers of neoprene in that they offer their own natural force feedback and pressure distribution.

Stretching piece of Zoflex video >>

The soft circuit that connects all the sensors to the power source, microchip and wireless communication device are a very prominent design aspect of the suit. Since for reasons of convenience and necessity they run visibly across the outside of the suit. Rather than seeing this as a problem we view it as one of the main design aspects.
Our (so far) preferred method of creating these conductive traces, which cover distances up to a max of 1.5 - 2 meters (when reaching all the way to the toes), is to work with thing strips of stretch conductive fabric. This method allows us to create stretchy fabric, comfortable, washable conductive connections that have a resistance of about 100Ohm/meter. Other options would be to use conductive thread, but since we work with stretchy materials and the stretchy stitch uses up a lot more thread that I directly sewn connection, we would reach much higher resistance using thread. Also, to be able to sew with a sewing machine, the full layout of the conductive traces on the suit would have to be clear before the suit is assembled and this has never been the case, since there are so many unknowns before the dancer first puts on the finished suit, and we find it much easier to be able to make alterations when the suit is worn and we can see exactly how it fits and where best to place the sensors.
After we have fused interfacing to a large piece of stretch conductive fabric from LessEMF (www.lessemf.com), we cut it into 5mm wide strips and are then able to fuse these to the suit with the heat from an iron. Once the traces are in place we then isolate them by covering them with a good layer of Aleene's Flexible Stretchable Fabric Glue (http://www.duncancrafts.com/). We have found this to be the best solution so far. It does not influence the conductivity of the trace, nor does it much impede on the stretchiness of the fabric. It dries within one day and after 10 days of curing time it can even be quite safely washed with lukewarm water and a mild detergent. The first puppeteer version did not use conductive fabric but we disguised the fragility of the thin wires, by feeding them through fabric tubes. The second Puppeteer version we used only the stretchy fabric glue to isolate the traces and then some baby powder on top to remove their initial stickiness, even once dry they like to stick to each other. For the last Puppeteer version we used the stretchy fabric glues to isolate as well as to mount strips of the same white fabric, so that they were less prominent, but still part of the texture and design of the suit.

In the last Puppeteer version we decided to make the sensor detachable so that we could exchange them and also it because quite a convenient way to attach them to the stretchy conductive circuit. We attached machine poppers to both sides of the sensors as well as to the ends of the conductive traces. We fabricated the sensors in three different lengths:
- Long for the knees, underarms and elbows
- Medium for the hips and shoulders
- Short for the wrists
- Customized shapes for the heels and toes

Certain parts of the circuit we simply did not want to risk washing or exposing to a lot of wear and tear. These components include: the power source batteries, the Microcontroller Arduino LilyPad (www.cs.colorado.edu/~buechley/diy/diy_lilypad_arduino.html), the XBee wireless module, the Multiplexers, as well as various small components such as voltage regulators, capacitors, resistors and wire connections. In the case of the first Puppeteer suit all of these components were anyway not affixed to the costume itself, because there was a wired connection. In the later costumes we chose to integrate these components in part of the costume that could be removed from the rest of the suit so that the sensors and conductive traces could be washed separately. For the second Puppeteer version we stored these components inside a little backpack type accessory that came out on top of the rest of the costume like a hood and plugged into the ends of the conductive traces coming from the sensors via some handmade lugs that we were could sew to. These we made from headers, wire and Shapelock http://www.shapelock.com
In the third Puppeteer version we integrated the components in the collar of the suit and the collar came right down the front of the suit, reaching the tops of the thighs and plugging into the ends of the conductive traces via metal poppers. In both cases we had access to the circuit by opening a zipper in the little backpack or the collar.

The design of the first Puppeteer costume was intentionally connected through a physical wiring, which the performer herself could disconnect, by unplugging herself. These were early stages and at first we used two Arduino USB boards to read the data from all 12 sensors, since the Arduino comes with 6 analog inputs. An improved version of this scaled down to one Arduino, using four of its six analog inputs directly and one of them in combination with an eight channel multiplexer.

The next Puppeteer suit became wireless, giving the dancer freedom to move around with no physical connection to the computer reading the sensor data. The only restrictions were those of the range of the Bluetooth module. We worked with the Sparkfun BlueSMiRF Gold module in connection with the Arduino LilyPad. The Bluetooth connection, though it worked, was very flaky so that for the third Puppeteer version we happily started working with the XBee, which has proved to be a great solution to all our wireless problems.

These sensors are easy to make, though time consuming and tedious. They have shown to deliver stable data, which we are able to work. They have not exhibited any drastic changes over time, even after washing with lukewarm water and a mild detergent they continue to react to pressure in the same range. When worn against the body during performance the sensor's range does tend to shift, not necessarily in size but in range location. Weather this is due to wrinkles in the fabric that contains them, or sweat or something that goes on inside the sensor we do not know.

The next steps include sensor development independently from creating the next Puppeteer suit.
Other ideas to explore include adding a kind of array of sensors to one costume, the idea being that if we have enough sensors we would be able to track poses and motion sequences that were unique because all the sensors could only ever be in a certain state if a certain pose or motion were being made.

Since the costume is tailored to fit a certain individual body size, the outcome of these projects has been specifically performance based. We are interested in expanding the design of the costume and creating things that can be worn and experienced by all. Here the planning of a modular system might make sense.

So far the Puppeteer sensors focus mainly on the bend of the joints of the human body, but we are interested in detecting other, subtler motions.

Aspects such as comfort, washability, durability and even sustainability (which is something we have not yet included) are things that we would like to continue to address in future.