HackerBox 0076: Biometrics

Welcome to HackerBox 0076. The adventure into biometric identification leverages face recognition, fingerprint scanning, and voiceprinting. The hardware used includes the ESP32 dual-core microcontroller with Wi-Fi, an OV2640 camera, an ultra-slim fingerprint scanner, and a MEMS microphone. TensorFlow Lite for Microcontrollers and similar software tools implement artificial intelligence and machine learning algorithms using neural network techniques for natural data modeling.

HackerBoxes is the monthly subscription box of electronics, computer technology, and hacker culture. Each HackerBox is a discovery box, which means all members await and enjoy a new surprise each month. Tech, toys, knowledge, and fun... It's like having a hacker convention, your birthday, and the first day of school - every month - right in your mailbox.

There is a wealth of information for current and prospective members in the HackerBoxes FAQ. Almost all of the non-technical support emails that we receive are already answered there, so we'd really appreciate it if you can take a few minutes to read the FAQ.

This Instructable contains information for getting started with HackerBox 0076. The full box contents are listed on the product page for HackerBox 0076 where the box is also available for purchase while supplies last. If you would like to automatically receive a HackerBox like this right in your mailbox each month with a $15 discount, you can subscribe at HackerBoxes.com and join the party!

A soldering iron, solder, and basic soldering tools are generally needed to work on the monthly HackerBox. A computer for running software tools is also required. Have a look at the HackerBox Workshops for basic tools and a wide array of introductory activities and experiments.

The most import thing you will need is a sense of adventure, hacker spirit, patience, and curiosity. Building and experimenting with electronics, while very rewarding, can be tricky, challenging, and even frustrating at times. The goal is progress, not perfection. When you persist and enjoy the adventure, a great deal of satisfaction can be derived from this hobby. Take each step slowly, mind the details, and don't be afraid to ask for help.

Biometric identifiers are the distinctive, measurable characteristics used to label and describe individuals. Biometric identifiers are often categorized as physiological characteristics, which are related to the shape of the body. Examples include, but are not limited to fingerprints, vein structures, facial recognition, DNA, hand geometry, iris recognition, retina, and scent. Similar, behavioral identifiers are related to the pattern of behavior of a person, including but not limited to typing rhythm, gait, signature, behavioral profiling, and voice.

Traditional means of access control general include token-based identification systems, such as a driver's license or passport, and knowledge-based identification systems, such as a password or personal identification number. Since biometric identifiers are unique to individuals, they are more reliable in verifying identity than token and knowledge-based methods; however, the collection of biometric identifiers raises privacy concerns about the ultimate use of this information.

(Wikipedia)

The ESP32-CAM Development Module does not include a USB bridge or USB connector, so connecting external USB support is required. In this case, we'll use the Micro CP2102 Serial USB Module (datasheet).

Solder the six pin header onto the Micro CP2102 Serial USB Module.

The OV2640 camera can be left off of the ESP32-CAM module for now.

Use five female-female jumper wires to connect the ESP32-CAM to the CP2102 USB Module as shown here.

Install the Arduino IDE software and set up the ESP32 board support package. This tutorial provides detailed instructions for that process. Currently, version 1.0.6 of the ESP32 package is preferred for the projects and libraries referenced in this Instructable. If a 2.x version is installed, it may be helpful to roll back.

In the IDE, select:

Tools > Board > ESP32 Arduino > AI Thinker ESP32-CAM

Also select the Tools > Port entry that appears when the CP2102 is connected to the PC

Be sure to have the "Programming Mode" jumper wire between I/O 0 and GND

Load the blink sketch: File > Examples > 01.Basics > Blink

Replace LED_BUILTIN with the value 4 in three different locations within the sketch.

Hit the arrow icon in the IDE to initiate compile and upload to the ESP32-CAM target.

If compiling completed but they upload is waiting for the bootloader with "....._____....._____" in the status window, hit the RST button on the ESP32-CAM. That should cause the upload to start.

Once the upload completes with "hard reset":

Remove one end of the "Programming Mode" jumper wire and hit the RST button on the ESP32-CAM.

This sketch will run, blinking the Camera's FLASH LED (connected to I/O pin 4).

Don't look directly at the FLASH... It's extremely bright.

This sequence needs to be used every time the ESP32-CAM is programmed:

1. connect "Programming Mode" jumper wire
2. press the IDE upload arrow icon
3. hit the RST button if upload is waiting "....._____....._____"
4. wait for upload to complete with "hard reset"
5. disconnect the "Programming Mode" jumper wire
6. open the serial monitor and set baud rate (when used)
7. hit the RST button

The ultra-slim fingerprint scanner features a tiny package usually found embedded within mobile devices such as smartphones or laptops. The scanner is based on an AS608 SOC from Synochip. The AS608 series chips feature high-performance 32-bit micro-processors based on the ARM Cortex-M. The chips feature a working frequency of 144MHz, 4KB of cache memory, 512KB of embedded Flash, 128KB SRAM, and extendible external SQI Flash up to 16MB.

Wire up the ultra-slim fingerprint scanner as shown using the included breakout cable. As usual, the colors of the wires are not necessarily meaningful. Instead, mind the relative pin ordering of the connections. Note that five of the breakout wires plug directly to the ESP32-CAM while one plugs onto the 3V3 pin of the Micro CP2102 Serial USB Module.

In the Arduino IDE, use Tools > Manage Libraries to search for and install two separate libraries:

1. EspSoftwareSerial (by Dirk Kaar)
2. Adafruit Fingerprint Sensor Library

Enroll (Learn) Fingerprints

Open: File > Examples > Adafruit Fingerprint Sensor Library > enroll

Make the following three modifications to the sketch...

Under the line to include <Adafruit_Fingerprint.h>, paste this line in :

#include <SoftwareSerial.h>

replace the line #define mySerial Serial1 with:

SoftwareSerial mySerial;

replace the line finger.begin(57600) with:

mySerial.begin(57600, SWSERIAL_8N1, 14, 15, false);

Compile and upload the sketch.

Before disconnecting the PROGRAM jumper wire and hitting rest,

Open the Serial Monitor and set it to 9600 baud

The interface printed to the Serial Monitor allows selection of ID # 1-127 to store separate fingerprint models.

Evaluate Fingerprints

Open: File > Examples > Adafruit Fingerprint Sensor Library > fingerprint

Make the same three modifications used for the enroll sketch.

Compile, upload, and reset.

Test the enrolled fingerprints and observe the output on the Serial Monitor.

Also try other fingerprints that were not enrolled.

How effective are the models created by the enrollment process to match the same fingerprints later?

How effective are the models to exclude false matches to other fingerprints?

Machine Learning is the process of taking in data (fingerprint sensor image) and associated answers (that image is my index finger) to create a representative model (learning) that can be used later (evaluated) to determine if new data (a new fingerprint sensor reading) is likely to be associated with the same answer (my index finger). The model is the key because it represents the learning obtained from the training process. In the fingerprint example, "enrollment" is how we train the models.

The INMP441 (datasheet) is a omnidirectional MEMS (micro electro mechanical system) microphone. The INMP441 provides high-performance, low power, digital-output, and a bottom-side sound port. The complete INMP441 solution consists of a MEMS sensor, signal conditioning, an analog-to-digital converter, anti-aliasing filters, power management, and an industry-standard 24-bit I²S interface. The INMP441 has a high SNR (signal to noise ratio) and a flat, wideband frequency response.

Solder and Connect Header Pins

Solder the six supplied header pins onto the microphone module. The header pins should extend from the same side of the PCB as the metal microphone housing. Sound enters the microphone from the other side of the PCB through the small gold circular opening.

Wire up the INP441 module as shown using five female-to-female jumper wires.

Record Audio via Microphone Module

Download ESP32_INMP441_RECORDING.ino from this github repo.

This sketch will record sound from the INMP441 microphone into a wave file. The wave file is created in the SPIFFS (Serial Peripheral Interface Flash File System) of the ESP32-CAM module. SPIFFS is a lightweight filesystem created for microcontrollers, such as the ESP32, that have a flash chip connected via the microcontroller's SPI bus.

Update the three lines of the sketch that define the I/O pin connections, like so:

#define I2S_WS 13
#define I2S_SD 14
#define I2S_SCK 16

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor to observe the progress of recording the wave file to the SPIFFS. This is where you provide your spoken words (or other sounds) to the microphone.

Retrieve Recoded Wave File from ESP32

Open the File System Browser sketch:

File > Examples > WebServer > FSBrowser 

Update the SSID and Password to a 2.4GHz Wi-Fi network that your PC also has access to:

const char* ssid = "wifi-ssid";
const char* password = "wifi-password";

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor to observe the IP address obtained by the ESP32.

Enter "192.168.168.168/recording.wav" in the URL field of your browser but replace the actual IP address from the Serial Monitor. This should either play the stored wave file from the ESP32 or cause the wave file to be downloaded to the PC where it can be played from download folder.

Obviously, the wave file should playback whatever speech (or other sounds) that you recorded from the microphone into the SPIFFS.

Recording, storing, and playing back sound may seem like a simple series of tasks. However, we can see from this exercise that there is quite a bit to it - especially with a small microcontroller. It can be done though, so let's keep in mind that the devices may be listening.

TensorFlow Lite for Microcontrollers is designed to run machine learning models on microcontrollers and other devices with only few kilobytes of memory. The core runtime just fits in 16 KB on an Arm Cortex M3 and can run many basic models. It doesn't require operating system support, any standard C or C++ libraries, or dynamic memory allocation.

In the Arduino IDE, use Tools > Manage Libraries to search for and install the library:

TensorFlowLite_ESP32

Open the example sketch...

File > Examples > TensorFlowLite_ESP32  > micro_speech_ESP-EYE

Switch to the tab for file "audio_provier.cpp" and adjust the I/O pin definitions, like so:

#define I2S_PIN_CLK    16
#define I2S_PIN_WS     13
#define I2S_PIN_DOUT   I2S_PIN_NO_CHANGE
#define I2S_PIN_DIN    14

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor and set it to 115200 baud to observe the sketch identifying utterances of "YES" and "NO".

The TensorFlow models used in the sketch were trained to apply to generic voice patterns using specific microphones and capture equipment. How accurate are they?

If you were to train your own TensorFlow models with your actual voice and audio hardware, using those models should be more accurate and possibly more sensitive to your specific voiceprint. Like a fingerprints, voiceprints can attempt to identify one particular person and exclude most, or all, others.

Here is more information on training your own models if you'd like to give that a try.

Install the OV2640 camera into the ESP32-CAM module. The camera connector on the ESP32-CAM is a white slot with a black tab. The black tab hinges forward and down away from the PCB and towards the white portion of the connector. This opens the camera connector. Once that is opened, the camera's flex connector can be inserted into the white slot with the lens facing out. Finally, the black tab is pressed back down up into the slot connector. Note that the lens has a protective cover sheet than can be peeled off prior to use.

In the Arduino IDE, open the sketch: 

File > Examples > ESP32 > Camera > CameraWebServer

Update the sketch with the SSID and Password for a 2.4GHz Wi-Fi network that your PC also has access to:

const char* ssid = "*********";
const char* password = "*********";

In the section of the sketch titled, "Select camera model", comment out the default line and uncomment the line for CAMERA_MODEL_AI_THINKER.

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor and set it to 115200 baud to observe the IP address.

Open that IP address in a browser and begin streaming from the webcam.

Experiment with the Face Detection and Face Recognition functionality.

How effective are they?

Theory of Operation

Here is a very nice tutorial on Additive Angular Margin Loss (aka ArcFace) which is the algorithm employed by the ESP Human Face Recognition Model.

Will A.I. usher in a new era of analog computing?

We hope you are enjoying this month's HackerBox adventure into electronics, computer technology, and hacker culture. Reach out and share your success in the comments below or other social media. Also, remember that you can email support@hackerboxes.com anytime if you have a question or need some help.

What's Next? Join the party! Live the HackLife! Get a cool box of hackable gear delivered right to your mailbox each month and enjoy a generous member discount. Surf over to HackerBoxes.com right now and sign up for your monthly HackerBox subscription.