Bionic Hand Controlled by OpenCV

Hello everyone, in this instructable I would like to guide you through the process of creating and mostly controlling your own bionic hand. This idea was created when I was mindlessly scrolling through instagram and came across short video where a person was using MPU6050 sensors to track the movements of his hand and display a 3D model of it on a screen, and since I worked with that sensor I though it would be a project, I would be capable of. Moreover I love connecting the world of programming with the real physical world, so I figured, why not take the measured data and transfer it to a real bionic hand and so it began. Later on I decided, that a more efficient way would be to use OpenCV instead of the MPU6050, partialy also because it was a way for me to explore and learn another skill.

Please note that the design of the hand is taken from inmoov, so a huge thanks goes to Gaël Langevin, who was kind enough to share his amazing work with the rest of the world.

Inmoov hand i2 and forearm

• 3D printer
• soldering equipment
• a little less then 1kg of filament (PETG or ABS or PLA)
• 3m fishing line (capable of holding around 20kg)
• 5x expansion springs 3/16″x1-3/4
• RTV silicone Ecoflex™ 00-10

Screws, nuts and bolts

• 10x M2x4 flat-head wood screws
• 5x M3x4mm countersunk head screw
• 4x M3x12mm flat-head wood screws
• 20x M3x12mm countersunk head screw
• 25x M3x16mm countersunk head screw
• 10x M3x20mm countersunk head screw
• 35x M3 nuts

Electronics

• 1x ESP32 38-pin DevModule
• 1x micro USB cable
• 5x linear hall sensor (49E)
• 5x disc magnet diameter 2.5mmx1mm
• 1x 16 vein ribon cable
• 5x 1k resistors
• 5x 2k resistors
• 6x servo motor (JX PDI-6225MG-300)
• 1x custom PCB (optional)
• 1x power supply (ideally 6V or 5V and around 100W since each servo draws up to 3A)

I uploaded all the needed STL files for the i2 hand below, though they are also available here. When printing, it is best to use a slightly higher infill (roughly 30%) in order to increase durability of the parts. Also as for the material, inmoov uses ABS, however if you do not have the setup for prining ABS reliably, PETG or PLA will also do the job.

Similarly as for the hand the files needed are below and also in the inmoov STL parts library. Beware that in the inmoov parts library are files for the orginal version of the inmoov robot. The hand is i2, thus you only need portion of the parts that there are for the forearm. Another important thing is that when printing Bolt_entretoise7 you are only going to need the middle bolt and clip (the other parts are for the old hand design).

You can additionaly print a little display stand, that I designed myself in fusion 360.

When assembling use the tutorials on inmoov for the hand i2 and forearm, which are quite detailed and give you all the need info.The initial part is realtively straight forward as it is just screwing together the whole design. The slightly tricky part is routing correctly the fishing line so that it is not tangled and installing the hall sensors that are in the fingertips.

For the fingertips it is important to use a really soft type of silicone, because the hall sensors have their uncertainties and thus the softer the better as the movement of the magnet inside it will be larger and more recognizable from the data. Once you have the sillicone part glued to the 3D printed one use it as a tool for adjusting how much the hall sensor is sticking out. After you have that set I highly advise to fix the hall sensor to the last part of the finger, otherwise there is a high chance that the hall sensors will shift a little after moving with the fingers, which will throw off the measurements.

In terms of the circuit the 16 servo driver module really makes a big difference, however there are some downsides to it too. There are two different versions of this driver, while they are almost the same there is a difference between the transistor used in the reverse polarity protection (for the capacitor) one version can withstand about 8A and the second one only about 0.5A, which is really a lot less then what the servos draw. Therefore it is best to either power the servos not through the driver module or make this small adjustment explained in this video and be careful when using a capacitor.

In terms of the hall sensors we need to use a voltage divider, because the output would be from 0V to 5V which the ESP32 can't properly read as it reads ADC only from 0V to 3.3V.

For the whole circuit you can either use a breadboard or prefferably a custom PCB (github link to my version).

Because servo motor and hall sensor is slightly different we have to test them.

The most important is testing the hall sensors as the values they measure will be what determines if the hand is pressing something hard enough. I advise plotting the data using Arduino IDE plotter to seewhere a value goes over the natural uncrtainty.

For this you can just use this very simple code snippet:

int hall = "Pin number your hall sensor is connected to";

void setup() {
 Serial.begin(115200);
 pinMode(hall, INPUT);
}

void loop() {
 Serial.println(analogRead(hall));
 delay(10);
}

OpenCV (python run in vscode)

In terms of the code running on the PC with the webcam we have to do two main things.

The first being the OpenCV tracking of the hand and its elements, from these elements we can then calculate the position of each finger.

The second thing is sending the data over the Serial port to the ESP32 so that it can manipulate the servos. This data can be relatively simplified, as we are not sending exact angle values but rather just the information, that a finger is either bent or not. So we can send five numbers of either 0 or 1 and a symbol at the end of that so that we can later recognise the index of each number.

To begin we need to import these libraries for our python code:

import cv2
import mediapipe as mp
import time
import serial

Then we need to create the class for processing the webcam data:

class HandDetector():
    # Constructor of the class with parameters for the measurement
    def __init__(self, mode=False, maxHands=1, detectionCon=0.5, trackCon=0.5):
        self.mode = mode
        self.maxHands = maxHands
        self.detectionCon = detectionCon
        self.trackCon = trackCon

        self.mpHands = mp.solutions.hands
        self.hands = self.mpHands.Hands()
        self.mpDraw = mp.solutions.drawing_utils

    # Function for finding and drawing the hand
    def findHands(self, frame, draw=True):
        imgRGB = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        self.results = self.hands.process(imgRGB)
        
        if self.results.multi_hand_landmarks:
            for handLms in self.results.multi_hand_landmarks:
                if draw:
                    self.mpDraw.draw_landmarks(frame, handLms, self.mpHands.HAND_CONNECTIONS)
        return frame
    
    # Function for finding each hand landmark and drawing its position
    def findPosition(self, frame, handNo=0, draw=False):
        lmList = []

        if self.results.multi_hand_landmarks:
            myHand = self.results.multi_hand_landmarks[handNo]

            for id, lm in enumerate(myHand.landmark):
                h, w, c = frame.shape
                cx, cy = int(lm.x * w), int(lm.y * h)

                lmList.append([id, cx, cy])

                if draw and id == 0:
                    cv2.circle(frame, (cx, cy), 15, (255, 0, 255), -1)
        return lmList

We need to define the main function:

def main():
    # The prevTime and currentTime are used to calculate the FPS later
    prevTime = 0
    currentTime = 0

    # Array for storing the info about the hand
    hand = [["Wrist", False], ["Index", False], ["Middle", False], 
            ["Ring", False], ["Thumb", False], ["Pinky", False]]

    # Initializing the Serial and opencv
    ser = serial.Serial(port="The name of the port the ESP32 is connected to")
    # I had to include the "cv2.CAP_DSHOW" because I had issues with the webcam loading on my linux machine
    cap = cv2.VideoCapture(0, cv2.CAP_DSHOW)
    detector = HandDetector()

    #MAIN LOOP OF THE CODE#

    # Releasing the stuff allocated for opencv
    cap.release()
    cv2.destroyAllWindows()

main()

And the main loop of the code:

    while (True):
      # Finding the hands and reading the position of hte landmarks
      ret, frame = cap.read()
      frame = detector.findHands(frame)
      lmList = detector.findPosition(frame)
        
      if len(lmList) > 0: 

            j = 1
            change = False
            # Loop which checks if the top of the finger is below the second most top
            for i in range(1, 6):
                if i == 1 and lmList[4][1] < lmList[3][1] and not hand[4][1]:
                    # In case that it is true it changes all the needed data
                    hand[4][1] = True
                    change = True
                    print(hand[4][0], hand[4][1])
                elif i == 1 and lmList[4][1] > lmList[3][1] and hand[4][1]:
                    hand[4][1] = False
                    change = True
                    print(hand[4][0], hand[4][1])
                elif i != 1:
                    if lmList[i*4][2] > lmList[(i*4)-2][2] and not hand[j][1]:
                        hand[j][1] = True
                        change = True
                        print(hand[j][0], hand[j][0])
                    elif lmList[i*4][2] < lmList[(i*4)-2][2] and hand[j][1]:
                        hand[j][1] = False
                        change = True
                        print(hand[j][0], hand[j][0])
                    if j == 3:
                        j += 2
                    else:
                        j += 1

            # If there has been any change in the state of the hand this code block will run
            if change:
                msg = ""
                # Converts the boolean values to 0s and 1s
                for i in range(6):
                    if hand[i][1]:
                        msg += "1"
                    else:
                        msg += "0"

                # Adds the ending symbol and sends the data over to the ESP32
                msg += '\n'
                print(msg)
                ser.write(msg.encode("Ascii"))

        # Calculates the FPS and displays it on the frame
        currentTime = time.time()
        fps = 1/(currentTime-prevTime)
        prevTime = currentTime
        cv2.putText(frame, str(int(fps)), (10, 70), cv2.FONT_HERSHEY_SIMPLEX, 3, (255, 0, 255), 3)

        # Shows what the webcam sees on a frame
        cv2.imshow("frame", frame)

        # If we press "q" it quits running the program
        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

Whole code OpenCV

import cv2
import mediapipe as mp
import time
import serial

class HandDetector():
    def __init__(self, mode=False, maxHands=2, detectionCon=0.5, trackCon=0.5):
        self.mode = mode
        self.maxHands = maxHands
        self.detectionCon = detectionCon
        self.trackCon = trackCon

        self.mpHands = mp.solutions.hands
        self.hands = self.mpHands.Hands()
        self.mpDraw = mp.solutions.drawing_utils

    def findHands(self, frame, draw=True):
        imgRGB = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        self.results = self.hands.process(imgRGB)
        
        if self.results.multi_hand_landmarks:
            for handLms in self.results.multi_hand_landmarks:
                if draw:
                    self.mpDraw.draw_landmarks(frame, handLms, self.mpHands.HAND_CONNECTIONS)
        return frame
    
    def findPosition(self, frame, handNo=0, draw=False):
        lmList = []

        if self.results.multi_hand_landmarks:
            myHand = self.results.multi_hand_landmarks[handNo]

            for id, lm in enumerate(myHand.landmark):
                h, w, c = frame.shape
                cx, cy = int(lm.x * w), int(lm.y * h)

                lmList.append([id, cx, cy])

                if draw and id == 0:
                    cv2.circle(frame, (cx, cy), 15, (255, 0, 255), -1)
        return lmList

def main():
    prevTime = 0
    currentTime = 0
    hand = [["Wrist", False], ["Index", False], ["Middle", False], 
            ["Ring", False], ["Thumb", False], ["Pinky", False]]


    ser = serial.Serial(port="COM3")
    cap = cv2.VideoCapture(0, cv2.CAP_DSHOW)
    detector = HandDetector()

    while (True):
        ret, frame = cap.read()
        frame = detector.findHands(frame)
        lmList = detector.findPosition(frame)
        
        if len(lmList) > 0: 

            j = 1
            change = False
            for i in range(1, 6):
                if i == 1 and lmList[4][1] < lmList[3][1] and not hand[4][1]:
                    hand[4][1] = True
                    change = True
                    print(hand[4][0], hand[4][1])
                elif i == 1 and lmList[4][1] > lmList[3][1] and hand[4][1]:
                    hand[4][1] = False
                    change = True
                    print(hand[4][0], hand[4][1])
                elif i != 1:
                    if lmList[i*4][2] > lmList[(i*4)-2][2] and not hand[j][1]:
                        hand[j][1] = True
                        change = True
                        print(hand[j][0], hand[j][0])
                    elif lmList[i*4][2] < lmList[(i*4)-2][2] and hand[j][1]:
                        hand[j][1] = False
                        change = True
                        print(hand[j][0], hand[j][0])
                    if j == 3:
                        j += 2
                    else:
                        j += 1

            if change:
                msg = ""
                for i in range(6):
                    if hand[i][1]:
                        msg += "1"
                    else:
                        msg += "0"

                msg += '\n'
                print(msg)
                ser.write(msg.encode("Ascii")) 

        currentTime = time.time()
        fps = 1/(currentTime-prevTime)
        prevTime = currentTime

        cv2.putText(frame, str(int(fps)), (10, 70), cv2.FONT_HERSHEY_SIMPLEX, 3, (255, 0, 255), 3)

        cv2.imshow("frame", frame)

        if cv2.waitKey(1) & 0xFF == ord("q"):
            break

    cap.release()
    cv2.destroyAllWindows()

main()

ESP32 (arduino IDE)

We can nicely take advantage of the fact that ESP32 is dual-core, because similarly as in the code for the PC we also need to be doing two main things.

The initial being the receiving of data from the PC. As previously stated the data is basically a string with a six digit binary number and an ending symbol. Moreover because this data is only being transmitted when a change in the state happens we can immediately assign these values (converted to true or false) to their respective variables. We can assign this task to the core 0 as the main loop runs on core 1.

The second thing is obviously the movement of the hand itself. For this we need to have loop running which constantly checks the state of the variables and if it has changed. Once a change occurs the servos are moved linearly by smaller steps. After each step we have to firstly check if the variable has not changed again and also we have to measure the values read by the hall sensors. If the values from the hall sensors are too high, meaning the magnet is too close to the core of the finger we also need to stop the movement of the servos.

Initially we will need the library for the servo driver and we will also include wire library for the I2C communication:

#include <Wire.h>
#include <Adafruit_PWMServoDriver.h>

After this we need to define the values of pulse length which are different for each type of servo so I highly advise you search them up for your specific servo or test them like this.

// Operating Speed of my Servo (6V): 0.21 sec/60°

#define SERVOMIN "Your value (mine was 70)" // This is the 'minimum' pulse length count (out of 4096)
#define SERVOMAX "Your value (mine was 510)" // This is the 'maximum' pulse length count (out of 4096)
#define SERVO_FREQ 50 // Analog servos run at ~50 Hz updates

Now we have to define the rest of the variables which we are going to use:

// Initializing servo driver object
Adafruit_PWMServoDriver pwm = Adafruit_PWMServoDriver(); 

// Index, Middle, Ring, Thumb, Pinky
// "state0" is the state the hand on the webcam is in and "state"
// is the stuff happening on the actual hand
bool state0[6] = {false, false, false, false, false, false};
bool state[6] = {false, false, false, false, false, false};

// Variable which indicates if there has been any change made to the state
bool change = false;

// Variables needed for reading the data from Serial
char sData;
String state;

// Variable for the hall sensor
// Index, Middle, Ring, Thumb, Pinky
// {pin, measured value, maximum value}
// ALL OF THE MAX VALUES WERE MEASURED BY ME THUS THEY WILL MOST LIKELY NOT BE SAME FOR YOU 
int hall[5][3] = {{26, 0, 2200}, {27, 0, 2400}, {14, 0, 2300}, {25, 0, 2200}, {12, 0, 2300}};

// Setting the index numbers of each motor
int wrist = 0;
int thumb = 4;
int index = 1;
int middle = 2;
int ring = 3; // IMPORTANT this motor will rotate in the oposite direction
int pinky = 5; // IMPORTANT this motor will rotate in the oposite direction

// Function for calculating the PWM based on the degree you want
int degToPwm(int degree) {
 return map(degree, 0, 320, SERVOMIN, SERVOMAX);
}

// Setting the degree thresholds used
int deg = degToPwm(75);
int deg1 = degToPwm(95);
int deg2 = degToPwm(85);
int startDeg = degToPwm(180);

After this we need to define the functions we are going to be using:

// Initialization of the task
TaskHandle_t recieveData;

// Function which reads the data from Serial
void recieveDataCode(void * parameter) {
 for(;;) {
  // Loop which runs when there is a message sent
  while(Serial.available()) {
   // Reading by each character
   sData = Serial.read();

   // If the character is the line ending symbol we know it is the end of the message
   if(sData == '\n') {
    // Loop for converting the string 0s and 1s to boolean
    for(int i = 0; i < 6; i++) {
     state0[i] = state.substring(i, i+1).toInt();
    }

    // Reseting the state temporary variable
    state = "";
    // Showing a change in state happened
    change = true;
    break;
   } else { // If the character is not the line ending symbol we add it to the temporary state
    state += sData;
   }
  }
  delay(10);
 }
}

// Function for actually moving the servos
void moveFinger(int fingerId, bool flex, int iteration) {
 // Because the ring and pinky motors move in opposite direction 
 // we have to check which motors we are moving
 if(fingerId != ring && fingerId != pinky) {
  // We also need to check if we want the finger to flex or straighten
  if(flex) {
   // Moreover the thumb moves a little less so we also check for that
   if(fingerId == thumb) {
    // Because we want to be able to control the movement throughout we have to 
    // divide it into smaller parts
    float fPwm = SERVOMIN + (float(103)*float(iteration))/float(130);
    // But we also have to make sure to convert back to int because float would
    // not be accepted by pwm function
    int iPwm = round(fPwm);
    pwm.setPWM(fingerId, 0, iPwm);
   } else { // If the finger is not the thumb we just move it
    pwm.setPWM(fingerId, 0, SERVOMIN + iteration); 
   }
  } else { // For the case that is retracting we have to just do the opposite
   if(fingerId == thumb) {
    float fPwm = deg - (float(103)*float(iteration))/float(130);
    int iPwm = round(fPwm);
    pwm.setPWM(fingerId, 0, iPwm);
   } else {
    pwm.setPWM(fingerId, 0, deg1 - iteration); 
   }
  }
 } else if(fingerId == ring || fingerId == pinky) { 
  // In the case of the ring or pinky finger we do again the same
  if(flex) {
   pwm.setPWM(fingerId, 0, startDeg - iteration);
  } else {
   pwm.setPWM(fingerId, 0, deg2 + iteration);
  }
 }
}

Now the only thing missing is the setup and loop functions:

void setup() {
 // Starting Serial on the same frequency as on the PC
 Serial.begin(9600);
   
 // Assigning the pinMode to all pins connected to hall sensor
 for(int i = 0; i < 5; i++) {
  pinMode(hall[i][0], INPUT);
 }
  
 // Setup and starting the servo driver
 pwm.begin();
 pwm.setOscillatorFrequency(27000000);
 pwm.setPWMFreq(SERVO_FREQ);

 delay(10);
  
 // Pinning the created task to core 0
 xTaskCreatePinnedToCore(
  recieveDataCode,
  "recieveData",
  10000,
  NULL,
  0, 
  &recieveData,
  0);
 delay(500);
}

void loop() {
 // Once there has been a change in the state this code block will run
 if(change) {
  // Looping firstly through the total steps of the servos
  for(int i = 5; i < 135; i += 5) {

   // Secondly through all of the hall sensors and reading the values
   for(int k = 0; k < 5; k++) {
    hall[k][1] = analogRead(hall[k][0]);
    // If the measured value is greater than maximum value we stop the movement
    if(hall[k][1] > hall[k][2]) {
     state1[k+1] = state0[k+1];
    }
   }

   // Thirdly through all the servo motors
   for(int j = 0; j < 6; j++) {
    if(state0[j] != state1[j]) {
     // If the state on the PC does not match the one on the esp32 we 
     // call the function for moving the respective finger
     moveFinger(j, state0[j], i);
    }
   } 
   // This delay is very important as it sets the speed of the movements
   delay(17);
  }

  // At the and we make the state variables equal again
  for(int i = 0; i < 6; i++) {
   state1[i] = state0[i];
  }
 }
  
 delay(100);
}

Whole code for ESP32

#include <Wire.h>
#include <Adafruit_PWMServoDriver.h>

#define SERVOMIN "Your value" 
#define SERVOMAX "Your value" 
#define SERVO_FREQ 50 

Adafruit_PWMServoDriver pwm = Adafruit_PWMServoDriver(); 

bool state0[6] = {false, false, false, false, false, false};
bool state1[6] = {false, false, false, false, false, false};

bool change = false;

char sData;
String state;

int hall[5][3] = {{26, 0, 2200}, {27, 0, 2400}, {14, 0, 2300}, {25, 0, 2200}, {12, 0, 2300}};

int wrist = 0;
int thumb = 4;
int index = 1;
int middle = 2;
int ring = 3;
int pinky = 5;

int degToPwm(int degree) {
 return map(degree, 0, 320, SERVOMIN, SERVOMAX);
}

int deg = degToPwm(75);
int deg1 = degToPwm(95);
int deg2 = degToPwm(85);
int startDeg = degToPwm(180);

TaskHandle_t recieveData;

void recieveDataCode(void * parameter) {
 for(;;) {
  while(Serial.available()) {
   sData = Serial.read();
   if(sData == '\n') {
    for(int i = 0; i < 6; i++) {
     state0[i] = state.substring(i, i+1).toInt();
    }
    state = "";
    change = true;
    break;
   } else {
    state += sData;
   }
  }
  delay(10);
 }
}

void moveFinger(int fingerId, bool flex, int iteration) {
 if(fingerId != ring && fingerId != pinky) {
  if(flex) {
   if(fingerId == thumb) {
    float fPwm = SERVOMIN + (float(103)*float(iteration))/float(130);
    int iPwm = round(fPwm);
    pwm.setPWM(fingerId, 0, iPwm);
   } else {
    pwm.setPWM(fingerId, 0, SERVOMIN + iteration); 
   }
  } else {
   if(fingerId == thumb) {
    float fPwm = deg - (float(103)*float(iteration))/float(130);
    int iPwm = round(fPwm);
    pwm.setPWM(fingerId, 0, iPwm);
   } else {
    pwm.setPWM(fingerId, 0, deg1 - iteration); 
   }
  }
 } else /*if(fingerId == ring || fingerId == pinky)*/ {
  if(flex) {
   pwm.setPWM(fingerId, 0, startDeg - iteration);
  } else {
   pwm.setPWM(fingerId, 0, deg2 + iteration);
  }
 }
}

void setup() {
 Serial.begin(9600);
   
 for(int i = 0; i < 5; i++) {
  pinMode(hall[i][0], INPUT);
 }
  
 pwm.begin();
 pwm.setOscillatorFrequency(27000000);
 pwm.setPWMFreq(SERVO_FREQ);

 delay(10);
  
 xTaskCreatePinnedToCore(
  recieveDataCode,
  "recieveData",
  10000,
  NULL,
  0, 
  &recieveData,
  0);
 delay(500);
}

void loop() {
 if(change) {
  for(int i = 5; i < 135; i += 5) {
   for(int k = 0; k < 5; k++) {
    hall[k][1] = analogRead(hall[k][0]);
    if(hall[k][1] > hall[k][2]) {
     state1[k+1] = state0[k+1];
    }
   }
   for(int j = 0; j < 6; j++) {
    if(state0[j] != state1[j]) {
     moveFinger(j, state0[j], i);
    }
   } 
   delay(17);
  }

  for(int i = 0; i < 6; i++) {
   state1[i] = state0[i];
  }
 }
  
 delay(100);
}