Put Yourself INSIDE a Video Game!

Hey gaming fans, I hacked the original Nintendo Punch-Out!! so that you control it with actual punches! It's super fun and it feels like you're actually INSIDE the game. Read on to learn how I modified a video game to control it with body movements instead of a controller, and play the finished game at RealLifePunchout.com

This project does not require anything beyond your home computer and a webcam.

This whole project is possible thanks to TensorFlow (Google's free AI framework) and specifically a TensorFlow model they trained called MoveNet. The goal of MoveNet is to look at each frame in a video and track your skeleton, basically. If you have a webcam on your computer, you can try out their super fast live in-browser demo.

In my project, I used MoveNet and the Javascript version of TensorFlow (called TensorFlowJS) to implement a skeleton tracker inside a web page. I added code that detects punches, dodges, and other movements you can make in the boxing video game I selected.

If your video game uses a console like the NES, Atari, N64, Sega, Super Nintendo, Playstation, XBox, and others... you normally can't play your games on your computer because the game cartridges only fit inside the console.

If you want to play these games on the computer, that's where an "emulator" comes in. An emulator is something that lets you play console games on your computer. The game I chose runs on the original Nintendo (NES or Nintendo Entertainment System) and luckily there are many emulators for this console. The one I used was called fceux. I wanted my game to run inside a web browser, so I used a cool tool called emscripten to compile fceux into javascript so that it runs inside a browser.

The games for your console system probably don't plug into your computer. So, how do you copy the game off the cartridge and onto your computer? The answer has something to do with ROM. You see, the game on your cartridge is saved in read-only memory which is also called a ROM. So if you copy the game off the cartridge, the resulting file is usually called a ROM for that reason.

The only legal way to obtain Nintendo ROMs that I'm aware of is to open your Nintendo and put a special circuit called a CopyNES in between your Nintendo's microchip and motherboard. This circuit has a USB port, and together with a program called USB CopyNES Blue it will allow you to copy a ROM to your computer. Remember, in order to respect Nintendo's copyright, you must never use a game cartridge and a ROM at the same time. If you only bought one cartridge, you can only play one copy of the game at a time.

Next, I wrote some code to send controller presses to the game according to what movements were detected by MoveNet. There was a problem however. Since it takes longer to move your whole body than it does to just press a button on a controller, the opponents moved too quickly for me to react in time. So, I had to slow down the opponents. The only way to do this is to hack the game using a process called disassembly.

When you disassemble something, your job is to start with the ROM which is the compiled and finished game, and reverse-engineer the original source code. It's one of the hardest things you can do in software, but thankfully there are many online resources to help. Once I disassembled the game I was able to slow down each opponent to make it a fair fight.

Now that the game is running inside the browser, and it's controllable using body movements, and the opponents aren't too fast, it's time to have fun and play the game! You can play the game I made for free by visiting RealLifePunchout.com

Enjoy!