Hummingbird Detector/Picture-Taker

We have a hummingbird feeder on our back deck and for the last couple of years I have been taking pictures of them. Hummingbirds are amazing little creatures, very territorial and their fights can be both hilarious and amazing. But I was getting tired of standing like a statue against the back of my house in order to take pictures of them. I needed a way to capture pictures without having to stand behind the house for long periods of time waiting. I know that I could have used a remote controlled shutter but I wanted pictures to be taken automatically without me having to be there. So I decided to make a device to detect the hummingbirds and take a picture automatically.

I always intended to use a microcontroller to do this. The microcontroller would be able to drive the camera shutter under software control. But the sensor to detect a tiny hummingbird was another thing. I could have used a motion sensor but I wanted to try something unique. I decided to use sound as the trigger.

The microcontroller I chose was a PJRC Teensy. The Teensy uses an ARM microcontroller, specifically, an ARM Cortex M4. The Cortex M4 contains the hardware to perform an FFT (Fast Fourier Transform) that would do the detection. PJRC also sells an audio board that allows you to use the Teensy to play music as well as record audio with an external input, or, a small microphone you can add to the board. My plan was to have the Teensy perform an FFT on the audio from the microphone.

An FFT is a mathematical formula/algorithm that transforms a signal from the time domain to the frequency domain. What this means is that it takes the time-sampled audio from the microphone and turns it into magnitudes of the frequencies that are present in the original wave. You see, any arbitrary, continuous wave can be constructed from a series of sine or cosine waves that are integer multiples of some base frequency. An FFT does the opposite: it takes an arbitrary wave and turns it into the magnitudes of the waves that would, if summed together, create the original arbitrary wave. An even simpler way to say this is, I planned to use software and the FFT hardware in the Teensy to determine if it 'hears' the wing flapping of a hummingbird at the frequency that the wing flaps occurs. If it does 'hear' a hummingbird, I will send a command to a camera to take a picture.

It worked! So, how did I do it, how could you do it and how could you make it even better?

First things first, I needed to figure out what frequency I would hear the hummingbird's wing flaps. To determine this, I used my iPhone. I attached the iPhone to a tripod and had it record slow motion video directly in front of the hummingbird feeder on our deck. After a period of time I removed the camera and I downloaded the video. I then watched the video looking for a hummingbird in front of the feeder. When I found a good sequence, I counted the number of individual frames it took for the hummingbird to flap its wings from one position all the way back to that same position. Slow motion on the iPhone is about 240 frames per second. I observed a hummingbird hovering in front of the feeder and I counted 5 frames for it to move its wings from the forward position to the back position and then return to the forward position. This is 5 frames out of 240. Remember, we hear a sound on each stroke of the hummingbird’s wing flaps (one on the forward stroke and one on the backward stroke). For 5 frames for a cycle or period, we can calculate the frequency as one divided by the period i.e 1 / (5/240) or 48 Hz. This means that when this hummingbird is hovering, the sound we hear must be twice this or about 96 Hz. The frequency is probably higher when they are flying and not hovering. It may also be affected by their mass but I think we can assume that most of the birds of the same species have about the same mass.

The Teensy (I used a Teensy 3.2) is made by PJRC (www.pjrc.com). The FFT will be calculated on a sample of sound. To acquire the sound, PJRC sells an audio adaptor board for Teensy (TEENSY3_AUDIO - $14.25). They also sell a small microphone that can be soldered to the audio adaptor board (MICROPHONE - $1.25). The audio adaptor board uses a chip (SGTL5000) that the Teensy can talk to over a serial bus (I2S). The Teensy uses the SGTL5000 to sample the audio from the microphone and digitize it, that is, create a set of numbers that represent the sound the microphone is hearing.

An FFT is just a fast version of what is called a Discrete Fourier Transform (DFT). A DFT can be performed on an arbitrary number of samples however, an FFT needs to have the samples stored in sets that are binary multiples. The Teensy hardware can perform an FFT on a set of 1024 samples (1024=2^10) so that's what we'll use.

An FFT usually produces, as its output, the magnitudes AND the phase relationships between the various waves represented. For this application we are not concerned with the phase relationships, but we are interested in magnitudes and their frequency.

The Teensy audio board samples audio at a frequency of 44,100 Hz. So, 1024 samples at this frequency represents a time interval of 1024/44100 or about 23.2 milliseconds. In this case, the FFT will produce as output, magnitudes that are integer multiples of the sample period of 43 Hz (again, 1/0.0232 equals about 43 Hz). We would want to look for magnitudes that are about twice this frequency: 86 Hz. It's not exactly the frequency of our calculated hummingbird wing flaps but, it's close enough as we'll see.

The libraries PJRC provides for the Teensy will process the samples and return an array of magnitude values. We will refer to each magnitude in the returned array as a bin.The first bin (at offset zero in the array of data we get back) is the DC offset of the wave. We can safely ignore this value. The second bin (at offset 1) will represent the magnitude of the 43 Hz component. This is our base period. The next bin (at offset 2) will represent the magnitude of the 86 Hz component, and so on. Each subsequent bin is an integer multiple of the base period (43 Hz).

Now this is where is gets a little weird. If we used an FFT to analyze a perfect 43 Hz sound then the FFT would return the first bin at some large magnitude and all of the rest of the bins would be equal to zero (again, in a perfect world). If the sound we captured and analyzed was 86 Hz then the bin at offset one would be zero and the bin at offset 2 (the second harmonic) would be some large magnitude and the rest of the bins would be zero, and so on. But if we captured the sound of a hummingbird and it was 96 Hz (as I measured on my one bird) then the offset 2 bin @ 86 Hz would have a magnitude of a slightly lower value (than the perfect 86 Hz wave would) and the bins around it (one lower and a few higher) would each have a diminishing non-zero value.

If the sample size for our FFT was larger than 1024 or if our audio sampling frequency was lower, we could make the resolution of our bins better (i.e smaller). But even if we changed these things to make our FFT bins 1 Hz multiples of the base period, we would still have to deal with this bin 'spillage'. This is because we'd never get a wing frequency that landed, always and exactly, on a single bin. This means we can't just base our detection of a hummingbird on the value in the offset 2 bin and ignore the rest. We need a way to analyze the data in a few bins in order to try and make sense of it. More on this later.

For my prototype hummingbird detector I used extra long male-male pins soldered to the pins in the Teensy. I did this so I could plug the Teensy into a small solder-less breadboard. I did this because I assumed I'd be making a lot of changes in the prototype and with the breadboard, I could change this and just jumper wires wherever I needed. I soldered female strips onto the bottom side of the audio board which allows it to be plugged on top of the Teensy. The microphone is soldered to the top side of the audio board (see pictures). More details on assembly can be found on the PJRC site:

(https://www.pjrc.com/store/teensy3_audio.html).

I have (well, my wife has) a Canon Rebel Digital Camera. There is a jack on the camera that lets you connect a manual remote shutter control. I purchased a manual remote control from B&H Photo. The cable has the correct jack to fit the camera on one end and is about 6 feet long. I cut the cable at the end near the button control box and I stripped back the wires and soldered them to three header pins that I could plug into the breadboard. There is a bare wire that is ground and two other signals: the tip is trigger (pink) and the ring (white) is focus (see the pictures). Shorting the tip and/or ring to the ground operates the shutter and the focus on the camera.

Using a jumper wire I ran a common ground from the Teensy over to an area where I could use it on the breadboard. I also connected the anode of an LED to pin 2 on the Teensy and the cathode of the LED to a resistor (100-220 ohms) to ground. I also connected pin 2 of the Teensy to a 10K resistor and the other side of the resistor I connected to the base of an NPN transistor (a 2N3904 found everywhere). I connected the emitter of the transistor to ground and the collector I connected to the white and pink wires from the cable that goes to the camera. The bare wire, again, was connected to ground. Whenever the LED is turned on by the Teensy, the NPN transistor will also turn on and it will trigger the camera (and the focus). See the schematic.

Because the Hummingbird's wing flapping frequencies likely don't go above a few hundred Hz, then we don't really need to record sound frequencies above, say, a few hundred Hz. What we need is a way to filter out only the frequencies we want. A bandpass or even lowpass filter would be great. Traditionally we'd implement a filter in hardware using OpAmps or switched-capacitor filters. But thanks to digital signal processing and the Teensy's software libraries, we can use a digital filter (no soldering needed...just software).

PJRC has a great GUI available that lets you drag and drop your audio system for the Teensy and audio board. You can find it here:

https://www.pjrc.com/teensy/gui/

I decided to use one of the PJRC-provided biquadratic cascaded filters to restrict the sound frequencies from the microphone (filter). I cascaded three such filters and set them for bandpass operation at 100 Hz. This filter will allow in to the system frequencies a little above and a little below the frequency we are interested in.

In the block diagram (see picture) i2s1 is the audio input to the audio board. I connected both audio channels to a mixer and then to the filters (the microphone is only one channel but, I mixed both so I didn't have to figure out which channel it was...call me lazy). I run the output of the filter to the audio output (so I can hear the audio if I want). I also connected the audio from the filters to the FFT block. In the block diagram, the block labeled sgtl5000_1 is the audio controller chip. It doesn't need any connections in the diagram.

After you do all this block construction you click on Export. This brings up a dialog box where you can copy the code that was generated from the block diagram and paste it into your Teensy application. If you look at the code you can see it is an instantiation of each control along with the 'connections' between the components.

It would take too much space in this Instructable to go over the software in detail. What I will try to do is highlight some of the key bits of code. But this isn't a very large application anyway. PJRC has a great video tutorial on using Teensy and the audio libraries/tools (https://youtu.be/wqt55OAabVs).

I started with some FFT example code from PJRC. I pasted what I got from the audio system design tool into the top of the code. If you look at the code after this you'll see some initialization and then the system starts digitizing audio from the microphone. The software enters the 'forever' loop() and waits for FFT data to be available using a call to the function fft1024_1.available(). When FFT data is available, I grab a copy of the data and process it. Note that, I only grab data if the largest bin magnitude is above a set value. This value is how I set the system's sensitivity. If the bins ARE above the set value then I normalize the wave and transfer it to a temporary array for processing, otherwise, I ignore it and keep waiting for another FFT. I should mention that I also use the microphone gain control function to adjust the sensitivity of the circuit (sgtl5000_1.micGain(50)).

Normalizing the wave just means that I adjust all the bins so the bin with the greatest value is set equal to one. All other bins are scaled by the same proportion. This makes the data easier to analyze.

I used several algorithms to analyze the data but, I settled on using just two. One algorithm calculates the area under the curve formed by the bins. This is a simple calculation that just adds the values of the bins across the region of interest. I compare this area to determine if it is above a threshold.

The other algorithm uses a constant array of values that represent a normalized FFT. This data is the results of a real (optimal) hummingbird signature. I call this a hedge. I compare the hedge data to the normalized FFT data to see if the corresponding bins are within 20% of each other. I chose 20% but, this value could be adjusted easily.

I also count how many times the individual algorithms think they have a match, meaning, think they hear a hummingbird. I use this count as a part of the hummingbird determination because, false triggering can occur. For example, when any sound is loud or contains the birds wing frequency, like clapping hands, you might get a trigger. But if count is above a certain number (a number I choose) I say it's a hummingbird.When this happens I turn on the LED to indicate we have a hit and this same circuitry triggers the camera via the NPN transistor. In the software I set the camera trigger time to 2 seconds (the time the LED and the transistor are on).

You can see in the picture how I (unceremoniously) mounted the electronics. I had the Teensy plugged into a breadboard which was stuck to a carrier board along with another (unused) Arduino compatible (an Arduino Zero I think). I wire-tied the whole thing to a metal awning pole on my deck (I also added strain relief to the cable running to the camera). The pole was right next to the hummingbird feeder. I powered the electronics with a small LiPo power brick that you might use to recharge a dead cell phone. The power brick had a USB connector on it which I used to run power over to the Teensy. I ran the remote trigger cable over to the Camera and plugged it in. I was ready for some bird action!

I set up the camera on a tripod near the feeder. I had the camera focused on the very front edge of the feeder and I set it to Sport Mode which takes several fast pictures when the shutter is pressed. With the shutter down time of 2 seconds I captured about 5 photos per trigger event.

I spent a couple of hours fiddling around with the software the first time I tried this. I had to adjust the sensitivity and the successive algorithm hit count. I finally got it tweaked and I was ready.

The first picture it took was of a bird that flew into the frame as if taking a high-speed bank turn like a jet fighter (see above). I can't tell you how excited I was. I sat quietly on the other side of the deck for a while and let the system work. I managed to record a lot of pictures but, I did throw away quite a few. It turns out, sometimes you just get a bird head or tail. Also, I got false triggers, which can occur. In total I think I kept 39 pictures. It took the birds a few trips to the feeder to get used to the shutter sound from the camera but they eventually seemed to ignore it.

This was a fun project and, it works. But, like most things, there's plenty of room for improvement. The filter could certainly be different (like a low pass filter or changes to the arrangement and/or parameters) and maybe that might make it work better. I am also sure that there are better algorithms to try. I'll be trying some of this in the summer.

I have been told there is open-source machine learning code out there...maybe the system could be 'trained' to identify hummingbirds! I'm not sure I'll try this but, maybe.

What other things could be added to this project? If the camera had a date/time stamper you could add that information to the pictures. Another thing you could do is record the audio and save it onto a uSD card (the PJRC audio board has a slot for one). The saved audio might be able to be used for training a learning algorithm.

Maybe somewhere a school of Ornithology could use a device like this? They might be able to glean information like feeding times, feeding frequency and, with the pictures, you might be able to identify specific birds that return to feed.

My hope is that someone else extends this project and shares what they make with others. Some people have told me that this work I have done should be turned into a product. I'm not so sure but, I would rather see it used as a learning platform and for science.

Thanks for reading!

To use the code I have posted you'll need the Arduino IDE (https://www.arduino.cc/en/Main/Software). You'll also need the Teensyduino code from PJRC (https://www.pjrc.com/teensy/td_download.html).