Sliding Puzzle 'Next Move' Suggesting Simple DL Model With ESP32 TensorFlow Lite

This project takes the game Sliding Puzzle (with simple 'next move' suggesting 'search-directed heuristic' option), adding to it the capability of suggesting 'next move' with a simple and naive DL model realized with ESP32 TensorFow Lite support. The Sliding Puzzle game is implemented for Arduino framework compatible microcontroller boards with aids from DumbDisplay to render the game remotely on your Android mobile phone. Specifically, ESP32 / ESP32S3 is the targe microcontroller board for this experiment, since it not only supports Arduino framework, it also supports TensorFlow Lite.

The DL model of this experiment is implemented with TensorFlow that I worked out by referencing to two of my previous experiments:

1. Trying Out TensorFlow Lite Hello World Model With ESP32 and DumbDisplay
2. Mnist Dataset -- From Training to Running With ESP32 / ESP32S3

In addition to an ESP32 / ESP32S3 microcontroller board, a few tools are assumed:

1. Python 3
2. Git
3. VSCode and PlatformIO -- reference on how they are used: A Way to Run Arduino Sketch With VSCode PlatformIO Directly
4. DumbDisplay Android app -- reference: Blink Test With Virtual Display, DumbDisplay

First, clone this project's source from the project's GitHub repository

Open the cloned directory ESP32SlidingPuzzle with VSCode, then open train_model.ipynb

Run all cells of the Jupyter Notebook train_model.ipynb.

1. If environment is not setup already, it should first prompt you to create / select an Python environment.
2. Create a virtual Python environment for the project
3. As the last step, when asked to install dependencies, make sure to select requirement.txt

The DL model is naively constructed with Keras like

With accuracy 0.868, the DL model appears not very good, but should be acceptable

The steps to build and upload the sketch is via PlatformIO (an extension of VSCode).

The configurations for developing and building of the sketch are basically written down in the platformio.ini file

Notice that you have choices of PlatformIO environments -- ESP32 and ESP32S3

Please choose the correct one according to the microcontroller board that you have.

In either case, the program entry point is src/main.cpp

Notice that

1. For ESP32S3, WiFi connection to DumbDisplay Android app is used. In such a case, you will need to provide necessary WiFi login credentials in a header file src/_secret.h that you need to create with content like

1. Otherwise, for ESP32, Bluetooth connection to DumbDisplay Android app is used, with the Bluetooth device name ESP32Sliding

Build and upload the sketch with VSCode menu item View | Command Palette -- PlatformIO: Upload and Monitor

1. For ESP32 which is using Bluetooth connectivity, you should see log entries with Serial Monitor (with baud rate 115200) like the second of the above screen shots
2. For ESP32S3 which is using Wifi connectivity, the log entries will be like third of the above screen shots. Notice that the IP to connect to the ESP32 board shown by the log entries.

To connect your Android phone to your ESP32 / ESP32S3 microcontroller board via DumbDisplay Android app, open the DumbDisplay Android app and make connection like

After connection, you should see the picture of the board drawn.

To start a game, double click on the board, to divide the board into grid of tiles

Double clicking on the board will randomize the board by 5 steps (5 random moves).

During game play, you can click Suggest for a suggested 'next move'. If you click Continuous, suggested 'next moves' are continuously made until either you disabled Continuous or the puzzle is solved.

There are three options for the 'next move' suggestion:

1. AI Only -- use the trained DL model for predicting 'next move' (the highest probability one)
2. AI+Search -- largely use the trained DL model for predicting 'next move'; however, if the top 'next move' probability is not high, fallback to use original "search* algo
3. Search Only -- use original "search" algo only

After every solving of the puzzle, 5 more randomize steps are added to randomization making the game a bit harder.

Any time if you want to reset / restart the game, double click on the Reset button.

The 'next move' suggestion is certainly not very accurate, specially when the board is randomized for many steps.

1. The naive DL model can certainly be improved. Notice that intrain_model.ipynb, the training data is randomized in reverse of how a board is randomize. Hence, it makes sense that the number of randomize steps affects the accuracy of the model specially in case the game is randomized for more steps than the model is trained. Here are the parameters for training the model:
2. round_count -- the number of rounds of random boards to generate; 10000 by default
3. random_step_count -- the number of random moves from "solved orientation" to "randomized orientation"; 20 by default
4. The "search" algo can be improved
5. The DL + "search" can be improved

Moreover, the board size can be bigger, like 5x5. In order to change the board size to 5:

1. Need to change tile_count of train_model.ipynb generate a sp_model_5.h for the sketch

1. Need to modify the sketch esp32_sliding_puzzle.ino

1. which controls

Interested friends are encouraged to try out and share how the 'next move' suggestion can be improved.

Hope that you will have fun with it! Enjoy!

Peace be with you! May God bless you! Jesus loves you! Amazing Grace!