Phenomenal Augmented Reality Allows Us to Watch How Things Are Watching Us!

In my childhood I discovered an interesting phenomenon: if I connected a light source to a sufficiently amplified television receiver, and waved the light around in front of a video camera, I could get the light to function as a 3D augmented reality display that would overlay the sightfield of the camera, as virtual information on top of physical reality. I grew up at a time when technology was transparent and easy to understand. Radios and televisions used transparent electronic lamps called "vacuum tubes" where you could see inside them and they revealed all their secrets. I then witnessed a transition into an era of opacity in which scientific principles were concealed inside integrated circuits with closed-source firmware. And while machines became more "secretive", we entered a new era in which they also began to watch us and sense our presence, yet reveal nothing about themselves (see a short essay I wrote about this metaphenomenon). I wanted to be able to sense sensors and see what they can see. So I invented something I called the PHENONAmplifier, a device that amplifies physical phenomenon in a feedback loop of revelation.

It worked by video feedback, and because of the feedback loop, it solved one of the most difficult problems in augmented reality: alignment between the real and virtual worlds. As a result I was able to make artistic "lightpaintings" or "light drawings/graphings" as scientific visualizations where the degree of visibility of each sampled point in space to a surveillance camera could itself be made visible. I called this "Metasensing", i.e. seeing what a camera can see (sensing sensors and sensing their capacity to sense). As a professor, I teach this to the students in my Wearable Computing and Augmented Reality class every year, but a number of people have attempted to reproduce this interesting scientific result and have had difficulty (it is somewhat difficult to get all the conditions right for the phenomenon to occur). Therefore I came up with a very simple way to teach and explain this phenomenon. Each student builds a very simple one-pixel camera, and a very simple 3D augmented reality system with a light bulb or LED to learn about this interesting effect. Once understood, this effect has many uses both artistically and scientifically. See for example IEEE Consumer Electronics, 4(4) Pages: 92 - 97, 2015.

Get the materials together and prepare them for assembly:

• Some black cardboard or other material to make a simple box camera;
• Glue, scissors, tape, or the like, to make the box (or black paint to paint an existing box black);
• A lens (or you can also just make a hole in the box to let light in, e.g. like a pinhole camera);
• Solderless "breadboard" or a circuit board, soldering iron, etc.;
• Wires and connectors;
• A suitable op amp such as TLC271 or TLC272;
• A transistor suitable to drive the light bulb or LED of your choice (I used 2N6058 to drive a 50 Watt light bulb, or 2SD 261 to drive a quarter-Amp LED);
• Heatsink, socket, and mica insulator if you're using the larger transistor;
• A light bulb or LED (the LED may also require a series resistor or the like, if no current limiting or ballast circuit is built-in);
• Resistors: one high valued resistor in the 1 to 100 megohm range works best (I used 47 megohms), and two low-valued resistors (e.g. 1000 ohms);
• A capacitor of suitable value to limit the bandwidth of the feedback (I used 6800pF);
• A photodetector, photodiode, solar cell, or the like, to use as the pixel (light sensor). Preferably this optical detector has sufficient surface area to provide a reasonable field-of-view for the 1-pixel camera.

Prepare the components by cutting off the leads to an appropriate length, especially the parts that are in the feedback circuit, e.g. the 47 Mohm resistor and the capacitor, as well as the photodiode.

To identify the polarity of the photodiode, connect it to a voltmeter. The anode terminal is the one that will provide a positive voltage when light is incident upon it. As shown in the picture, you can see more than 0.3 volts under illumination of a typical desk lamp. Sometimes the polarity is indicated by the lengths of the leads, so you might want to mark the positive (anode) lead with a red sharpie, as indicated, prior to trimming the leads to a shorter length.

Make a simple one-pixel camera. You can learn a lot from a 1-pixel camera because it forces you to think fundamentally about what a pixel is and what it measures or senses. This is a great way to learn about a camera in its simplest form. This is also what led me to the invention of HDR imaging (mathematically turning a camera into an array of light meters).

The camera can be made in a black cardboard box, or take an existing housing and paint it black inside, and affix a lens to the front of it. Focus the lens so that it forms an image on the place where the sensor is. The sensor is a photodiode.

So you will end up with a black box that has a lens on the front to allow light in, and two wires coming out, e.g. red and black, red for positive and black for negative.

If you have a dark enough room (and some black cardboard on your workbench, surroundings, etc.) you may also omit the box and just have a lens that's the right distance from the sensor, along with a couple of pieces of black cardboard on either side, to keep most of the light out of the space between the lens and the sensor. In one of the pictures you can see that I have merely leaned a piece of black foam rubber against the lens to block light from getting to the sensor without first going through the lens. In this case I used an HP plotter to move an RGB LED back and forth in a known plane of motion, and since I know the light will only be coming from the sides (not the top or bottom) I only need 2 sides to the "box".

In fact you can perform a quick test of the system with just a lens in front of the sensor (no box around it). See picture above (lens glued to side of heavy battery sitting on the workbench, so it can be focused by moving the battery forward and backward).

Next you will build the amplifier that the camera connects to. The camera connects to the amplifier and the amplifier connects to the light source.

Note also that the camera sensor (photodiode) should be reverse-biased (e.g. reverse-connected). The red wire (positive, i.e. anode) will be connected to the ground pin of the amplifier and the black wire will be connected to the amplifier's input.

Now let's build the amplifier!

You will now learn how to build a very simple amplifier with massively high gain, such as to be able to exhibit an example of the "Phenomenological Video Feedback Effect" that allows an otherwise invisible physical phenomenon to be made visible, through electro-optical feedback with a moving light source.

The amplifier accepts input from the camera, and drives the light source. When the light is moved, it makes the phenomenon visible through a combination of two concepts:

• Video feedback. Instead of the usual fractal patterns on a TV screen, we have only a one-pixel display. This gives rise to a reversal of how video feedback usually works: rather than moving a camera around and pointing it at a stationary screen, we move our "screen" (in this case our light is a 1-pixel "screen") around while the camera remains stationary; and
• Persistence of Exposure (PoE). Persistence of Exposure can take place within another camera (e.g. set to long-exposure) or within the human eye itself, which functions very much like a camera.

Ordinary amplifiers that amplify voltage are unstable or unreliable at extremely high gains, and require many stages to get extremely high gain, but there are other kinds of amplifiers in which you can obtain an extremely high gain from one single stage! There are four main categories of amplifiers:

1. voltage amplifiers. Voltage input gets amplified to voltage output;
2. current (amperage) amplifiers. Current input gets amplified to current output;
3. those that convert voltage to current; and
4. those that convert current to voltage.

Amplifiers are characterized by their transfer function, i.e. "h" = "output" divided by "input".

h=output⁄input

So an amplifier (type 3) that converts voltage to current is called a "transconductance" amplifier, since the units of "h" (units of the output/input) = amps/volts = conductance. Vacuum tubes are examples of this kind of amplifier: grid voltage is converted to plate current.

The amplifier we'll use in this Instructable is the 4th fourth kind of amplifier. This is the one that I drew out on the chalkboard above (ECE516 class 2016 Jan 28). It is called a "TransImpedance Amplifier" or "TIA". It converts current to voltage, so the units of its transfer function "h" (units of output/input) are volts divided by amps, i.e. ohms (units of impedance).

In our application of the TIA, we're converting the photocurrent (current from the photodiode) to voltage. In this configuration the amplifier can operate reliably at extremely high gain with only one stage required! Here the gain is 47,000,000 volts per amp. Obviously if you put one amp in, you won't get 47,000,000 volts out, because it will saturate at the 12 volts or so supply voltage. But if you have 200 nanoamperes of photocurrent, you'll get about 9.4 volts or so output. Typically I use an 11-position rotary switch to select from the following resistors in a 1-3 sequence: 10k, 30k, 100k, 300k, 1m, 3m, 10m, 30m, 100m, 300m, and 1gigohm.

Connect the camera to the input of the amplifier as shown above. Additional pictures, including the 46 still images that correspond to the .gif animation above, are available here (link). Connect a voltmeter to the output of the op amp (pin 6 if you're using a TLC271). Cap the camera lens (e.g. with black tape that's black all the way through, or heavy black cardboard). You should see a very low near-zero voltage on the meter. Uncap the lens and let some light in. The output should vary linearly with the quantity of light. You can leave the meter connected, allowing you to use this apparatus as an accurate light meter.

You can vary the gain by varying Rf, depicted in the chalk drawing above. As the gain increases the circuit can become unstable. Use a feedback capacitor, Cf, in parallel with Rf, to reduce the gain at high frequencies, while keeping it high at low frequencies. Experiment with and without the capacitor to see its effect.

I prefer not to use a potentiometer (variable resistor) for Rf, because the wires or leads to and from it, along with its larger electromagnetic cross-section, can pick up stray noise signals.

Once you have confirmed that your 1-pixel camera is functioning as a light meter (which is the general philosophy of comparametric analysis), connect it to the transistor, in common emitter configuration as shown in the drawing. A base resistor for the transistor, Rt, limits the current into the base. I chose the 2SD261 for driving an LED because it has good current gain, and I didn't need a high frequency transistor. I found a surplus version that came with a nice heatsink. Double-check the pinouts. For the NEC D261 version of this transistor, the collector is in the middle (unusual for this TO-92 package where base is usually in the center). The one available on Digikey has a different pinout (on the Digikey version the base is in the middle).

Rb is a bias resistor. The purpose of this resistor is to keep the light glowing slightly even in complete darkness. This is required to initiate the metasensing optical feedback effect.

You may want to experiment with this value, or even substitute a potentiometer to be able to vary the "bias".

Now let's do some AR!

In a dark room, you should be able to wave the light around. I like to use an LED because it responds quickly and allows you to wave the light around more quickly. Note that if the room is not dark enough, the LED might go to full brightness even when it is not in view of the camera. In this case you might have to decrease the gain from 47,000,000 ohms to something less. Try 1,000,000 ohms for example, until the system works properly, and then increase the gain a little bit to see how much gain you can get away with for a given light level.

If you have a darkroom, that's a good place to begin experimenting with what we call "Phenomenal Augmented Reality" (augmented reality of physical phenomenology).

With your eyes adjusted to the dark you should be able to observe the effect. Try not to let your eyes follow the light. You might want to affix a separate small indicator light, or stare at something like a darkroom timer with glowing hands, and as you wave the light back and forth it can be quite amazing and educational to see the phenonenological overlays imprinted on your retina by Persistence of Exposure.

If you have a photographic camera such as a DLSR, that has a "B" ("Bulb" or long-exposure) setting, try photographing the trails of light from the apparatus. A combination of flash and long exposure can also be quite amazing as the flash will make visible the system under test, while the long exposure will make visible the otherwise invisible virtual material (e.g. the sightfield of the camera under test). In this case you have 2 cameras: your test-camera (the one that's exhibiting the phenomenon) and your photographic camera (the one that's photographing the other camera that's exhibiting the phenomenon).

I'd love to see what you come up with!

Now that you've built a simple example of Phenomenal Augmented Reality, this opens the door to understanding a whole new world!

Try tracking the light source with Meta Spaceglasses (Metaglasses) and you can capture the sight into the eyeglasses and track and make a map of a camera and walk around and see its sightfield from multiple angles. See for example the "abakographic principle" applied to Metaglasses. You can also build robotic systems that implement the abakographic video feedback AR metasensing principle. Back in the 1970s I built something I called the "Sequential Wave Imprinting Machine" that allowed me to see radio waves, as well as surveillance camera sightfields by receiving the signals from cameras, through wearable computing and a wearable antenna array. In building these early amplifiers I used triacs to drive even bigger lights, such as arc lamps or other devices up to 12,000 watts (see above example in which the light is bright enough to melt snow to create a snow sculpture in which a surveillance camera's gaze burns through the snow).

Have fun and experiment safely, especially when melting snow or steel with your camera's gaze!