A Touchless Input Device for Interactive Kiosks in Post-COVID World

The proposed solution is intended to decrease physical contacts while using embedded public computers, such as interactive kiosks, ATMs, information terminals, check-in kiosks, ticketing kiosks, wayfinding kiosks, etc. Most of them have a touchscreen interface which is a serious problem during the pandemic. Sometimes people are unable to avoid interacting with such devices since it may be the only way to make use of a particular service.

The idea is to make customers use their smartphones as universal input controls for embedded public computers. The device we are going to build is supposed to be a bridge between a smartphone and an interactive kiosk. It acts like a standard mouse for an embedded computer and receives commands from a smartphone. We’ll also develop a mobile app for interacting.

Most interactive kiosks and public embedded computers are in fact single-board computers with GNU/Linux, Android or Microsoft Windows onboard. They likely have USB input devices support. To make a solution universal, a USB mouse emulation seems to be the most suitable option.

The existing post-COVID touchless solutions often address very specific public computers and require implementing some changes into embedded systems’ software. What makes the proposed solution different:

1. A universal solution: it should work on any embedded device under any operating system without additional software or patches, given it supports a standard USB mouse.

2. An easy-to-use solution: an interactive kiosk owner plugs our device with uploaded software and attaches a sticker with QR code or information on how to use contactless input.

3. A low cost solution: users do not need to carry any special devices. They use client software on their own cell phones. A cell phone screen is used as a touchpad to navigate through kiosk’s menus.

🔗 If you are interested in building HID devices with Arduino, you might also like my BadUSB tutorial. I have already explained some essentials of Bluetooth communication between Arduino board and a mobile App in another instructable. So you might be interested to check it out too.

Hardware

1. A Bluetooth module. The most common are HC-05 and HC-06. For this project there is no difference between them.
2. A microcontroller with USB support and serial interface to communicate with HC-0x module. The easiest way would be to pick some Arduino Leonardo-like board with ATmega32U4. But I am going to use a Digispark-like board with ATtiny85. It is cheaper and smaller. But it has neither hardware USB nor hardware serial interface. And it’s going to be an interesting technical challenge.

Software

1. Arduino IDE.
2. DigisparkMouse and Digispark_SoftSerial-INT0 libraries.

There are a lot of tutorials on how to setup DigiSpark Attiny85 board support in Arduino IDE. On Instructables too. So check them out if you face some difficulties.

As I mentioned, ATiny85 does not have hardware USB and Serial support. The USB communication on Digispark boards is implemented with pure software. There is DigisparkMouse library in DigistumpArduino distribution which might be used to emulate a USB mouse. And there is DigisparkSoftSerial library for software serial interface. But, as I found out very soon, it is impossible to use them simultaneously since they use the same hardware interrupt. After looking for a bypass, I finally found Digispark_SoftSerial-INT0 library which uses INT0 interrupt and might be used with DigisparkMouse.

You must have already installed DigisparkMouse when you added Digispark board support to Arduino IDE. Now you should install Digispark_SoftSerial-INT0. Just copy the DigisparkSoftSerial_INT0 folder into Arduino libraries folder.

Since P3 and P4 pins are used by USB and P5 is Reset, we'll use P2 as UART Rx and P1 as Tx. So only P0 is not used in this project.

The attached sketch is pretty much straightforward. We just wait for character commands from Bluetooth and move the mouse. There are five commands:

l — left;

r — right;

u — up;

d — down;

c — click.

If you need a right click, you could easily implement it. Touch interfaces usually don’t use it.

Please read the comments for further details.

#include <SoftSerial_INT0.h> //Software Serial library
#include <DigiMouse.h> //Software USB mouse library
#define P_RX 2                       // Receiver pin for SoftSerial 
#define P_TX 1                       // Transmitter pin for SoftSerial 

SoftSerial BLE(P_RX, P_TX);          // initializing SoftSerial

void setup()
{
   BLE.begin(9600); // starting serial port
   DigiMouse.begin(); // starting mouse
}

void loop()
{
   static char command; // a character command from Bluetooth

   if(BLE.available() >= 1) // if there is something to read
     {
       command = BLE.read(); // read a command
       switch(command){
         case 'l': DigiMouse.move(-2,0,0); // l - move left
         break;
         case 'r': DigiMouse.move(2,0,0); // r - move right
         break;
         case 'u': DigiMouse.move(0,-2,0); // u - move up
         break;
         case 'd': DigiMouse.move(0,2,0); // d - move down
         break;
         case 'c': left_click(); // c - left mouse click
         break;
       }
     }

   DigiMouse.update(); // Keep on USB connection
}

void left_click() //mouse click
{
   DigiMouse.setButtons(1<<0); // Click
   DigiMouse.delay(10);
   DigiMouse.setButtons(0); // Unclick
}

Register on http://ai2.appinventor.mit.edu or sign in with your Google account and create new project.

Drag a VerticalArrangement onto app's main screen and make it fill all available space. It will align all the controls vertically.

Drad a ListPicker onto your Layout. This widget will be used for Bluetooth device selection.

Drag a BluetoothClient onto your screen. This invisible object will be used to control Bluetooth communication.

Drag a Button just below the ListPicker. Make it fill all the screen width and call it "Connect".

Drag a Label just below your "Connect" Button. The label will be used for connection status indicating. Make it red and give it a text value "Disconnected".

Drag a Canvas and make it fill all the free space. We'll use it this widget as a touchscreen surface.

Drag a Timer to your form. This object will be invisible as the Bluetooth object you dragged earlier. The timer will be used to send data with equal time intervals. Set the intervas in milliseconds. It should not be too short or too long. 50ms is a good guess to start with. You may try change it later.

Switch to Blocks mode and add the blocks like on the attached picture. If you find it out difficult to understand, please check out my earlier instructable regarding Bluetooth communication between Arduino and App Inventor mobile app.

Now you should be able to upload your app to Android device. The easiest way is to generate QR code and download apk file. First go to Anrdoid Bluetooth settings on your smartphone and allow connecting to HC-06. If it asks password, the default one is "1234". Now you could run your app and open devices list. Make sure your HC-06 module is available there.

After connecting you should be able to control your mouse pointer from the app.

This project is nearly on the borderline of ATtiny85 board’s capabilities. We have used 4 out of 5 outputs, about 80% of 6Kb flash memory and half of RAM. And I kind of like it. ATmega32U4 would have been an overkill. However this is just a proof of concept of an affordable and easy to build post-COVID input technology to work together with existing touchscreen devices instead of replacing them. If you would like to implement some authorization techniques and encrypted connection, you’ll probably need another microcontroller. In fact I intentionally tried to keep the Arduino sketch and the mobile app as compact as possible, yet sufficient to demonstrate the concept. But you could easily extend them. A custom PCB would also be advantageous at the next stage of prototyping.