FPGA Robotics Vision for Anyone

Are you like me interested in technology and always wanted to use one of those heavy chips called FPGA's? Are you interested in vision and robotics?

My goal was to bring FPGA vision to the people, accessible to everyone. I decided to build my own basic vision system. Not based on some FPGA manufacturers expensive IP and hardware. No! Just by using a camera, LCD display that can be found anywhere on Ebay or chinese webshops.

I used an affordable FPGA from Xilinx on a VERY affordable $31 development board...the only thing required was lots of VHDL code!

Why did I use an FPGA to do this? Most people use a Microcontroller right? And there are options out there that give your robot out-of-the-box basic vision for a fair price like the Pixy CMUCAM5. I own one, it is based on 2 ARM microcontrollers. I used an FPGA because of its tremendous power to do all kinds of tasks in parallel making it very useful for a task like Vision. And because I think there are more peope like me who want to use an FPGA but did not know where to start.

The result is a real-time 30fps imaging system with basic color detection. This is called "tresholding". What it does is compare the QVGA images coming from the camera at 30fps with a pre-defined "mask" for the R, G and B components. The system "spots" the pre-defined color and shows this on the 320x240 (QVGA) LCD. Pixels complying with the "mask" condition are shown white and those who not comply stay black.

Right now (in the VHDL code) this "mask" is a bright almost white color. This means the R,G and B components are almost at their maximum.

I also implemented "template matching". This means you can compare a pre-stored camera image to an external FIFO memory (design of the board is included) with the real-time images coming from the camera and see the differences real-time on the LCD! I will bring this later to Instructables.com.

Feel free to use my system to experiment and maybe implement your own algorithms.

The Bill of Materials is not so large. I used an FPGA board with a Xilinx FPGA and it was a Spartan-6 (those are affordable).

• You need to go to http://www.xilinx.com and register an account. After that download the Xilinx ISE webpack Version 14.7
• Download the ZIP file attached to this step, it contains the project code including all VHDL files
• a real nice guy from China I know called “Adam” sells a good and affordable FPGA board Xilinx-XC6SLX9-based board like the Xilinx-XC6SLX9-Spartan-6-FPGA-Core-Board-Nano-Size on ebay
• You will also need a Xilinx programmer, the original is very expensive but a clone can be obtained for $31,-
• An 2.8" TFT screen with ILI9325 chipset like this one (I used it in 16bit mode RGB565 so be sure that the one you buy supports this)
• An OV7670 camera board with NO fifo mounted (bare-camera board)
• A set of so-called Dupont cables like these to wire it all together

I have tried other OV7670 modules with screw-on lenses (still without FIFO) but have seen issues with those: they gave me green colored images.

That is why I used the version with the miniature camera (no focus lense) and ZIF socket. It costs slightly more but gave me no problems.

The installation of Xilinx ISE is straight forward. Switch off your antivirus, it speeds up the installation.

The setup of the project should not be required if you just open the project from the ZIP file I added to Step 1.

However might there be any issue please refer to the User Manual of the Ebay Xilinx board I recommended in Step 1. I attached it to this step.

It describes in great detail how to install ISE and also how to setup your projects tailored to this FPGA board.

I presume in this step that you use the same Spartan 6 board from Ebay which I mentioned earlier in Step 1. Connections present on the Spartan board are not shown (CLK in, leds, SW1, SW2 etc).

Connections mentioned here you need to make between FPGA board, TFT and Camera using the Dupont wires.

In the Xilinx project folder the .UCF file can be found which contains the complete FPGA board connections , use this if you have another Spartan6 board or you want to make your own board.

See the labels with pin nrs on the Spartan board for pin numbers (e.g: P95)

Camera databus

OV7670_DATABUS[7] TO SPARTAN6 XC6SLX9 FPGA PIN P95

OV7670_DATABUS[5] TO SPARTAN6 XC6SLX9 FPGA PIN P98

OV7670_DATABUS[6] TO SPARTAN6 XC6SLX9 FPGA PIN P94

OV7670_DATABUS[3] TO SPARTAN6 XC6SLX9 FPGA PIN P100

OV7670_DATABUS[4] TO SPARTAN6 XC6SLX9 FPGA PIN P97

OV7670_DATABUS[2] TO SPARTAN6 XC6SLX9 FPGA PIN P99

OV7670_DATABUS[1] TO SPARTAN6 XC6SLX9 FPGA PIN P102

OV7670_DATABUS[0] TO SPARTAN6 XC6SLX9 FPGA PIN P101

Camera control signals

OV7670_HREF TO SPARTAN6 XC6SLX9 FPGA PIN P87

OV7670_PIXELCLOCK TO SPARTAN6 XC6SLX9 FPGA PIN P93

OV7670_SCL TO SPARTAN6 XC6SLX9 FPGA PIN P85

OV7670_SDA TO SPARTAN6 XC6SLX9 FPGA PIN P84

OV7670_VSYNC TO SPARTAN6 XC6SLX9 FPGA PIN P88

OV7670_XCLK TO SPARTAN6 XC6SLX9 FPGA PIN P92

TFT screen databus

TFT_DATA[15] TO SPARTAN6 XC6SLX9 FPGA PIN P27

TFT_DATA[14] TO SPARTAN6 XC6SLX9 FPGA PIN P29

TFT_DATA[13] TO SPARTAN6 XC6SLX9 FPGA PIN P24

TFT_DATA[12] TO SPARTAN6 XC6SLX9 FPGA PIN P26

TFT_DATA[11] TO SPARTAN6 XC6SLX9 FPGA PIN P22

TFT_DATA[10] TO SPARTAN6 XC6SLX9 FPGA PIN P23

TFT_DATA[9] TO SPARTAN6 XC6SLX9 FPGA PIN P17

TFT_DATA[8] TO SPARTAN6 XC6SLX9 FPGA PIN P21

TFT_DATA[7] TO SPARTAN6 XC6SLX9 FPGA PIN P15

TFT_DATA[6] TO SPARTAN6 XC6SLX9 FPGA PIN P16

TFT_DATA[5] TO SPARTAN6 XC6SLX9 FPGA PIN P12

TFT_DATA[4] TO SPARTAN6 XC6SLX9 FPGA PIN P14

TFT_DATA[3] TO SPARTAN6 XC6SLX9 FPGA PIN P10

TFT_DATA[2] TO SPARTAN6 XC6SLX9 FPGA PIN P11

TFT_DATA[1] TO SPARTAN6 XC6SLX9 FPGA PIN P8

TFT_DATA[0] TO SPARTAN6 XC6SLX9 FPGA PIN P9

TFT screen control signals

CS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P32

EN_BACKLIGHT TO SPARTAN6 XC6SLX9 FPGA PIN P35

ILI9325_READY TO SPARTAN6 XC6SLX9 FPGA PIN P41

RD_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P33

RESET_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P40

RS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P30

WR_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P34

VHDL code is not like C code. Actually VHDL is a description language, with this language you can "describe" how the FPGA chip should behave. It's basically like drawing a schematic but now you do it in words.

Toplevel

Have a look at the picture attached to this step, this is called Toplevel. On the toplevel all individual code blocks can be seen and I will describe shortly what their functions are.

POWER_ON_RESET_CONTROLLER

This block takes care that all the others are properly reset when the power is applied to the system. It does so by waiting an amount of time using a state-machine and while waiting it makes it's output NOT_POR_RESET low. This signal is connected to all other blocks and resets them / prevents them from starting until this reset pulse is over (going to HIGH level).

Blinkled

Just my way of hello-world, I use this to let my board indicate its alive (by blinking leds)

clock_divider

Divides the clock by 2 and feeds this to the OV7670 because the OV7670 cannot use the 50MHz from the fpga board (too high).

datasplitter

Acts as a bidirectional databus buffer with 3state option. Not really used yet, the TFT is only written to but it is possible to also read from the TFT registers.

ili9325_16bitcameradata

This block first configures the TFT it is connected to (right after power on reset is finished). For instance it sets the TFT into RGB565 mode. After that it goes into write mode and all it does is constantly update the TFT view (showing either camera images or tresholded images. When this block is done configuring it signals to the next block that it can start it's configuration:

OV7670_sccb_initialisation

This block took a lot of work. The Omnivision camera (OV7670) needs to be configured after power-up. For example we need to set it's output to RGB565 mode (meaning the R-factor of a pixel is 5 bits, G is 6 and B is 5 again). I chose this to make it match with my ILI9325 TFT which has the same feature. Sccb is very similar to I2C (I-square-C, protocol invented by Philips a long time ago). But it has its differences which took me many many hours to get right!

image_tresholding

Last but not least. This is where the actual magic happens. The block gets images from the OV7670 camera (driven by the camera's HREF and VSYNC lines which together indicate a start + end of frame (image)). Inside the block the comparison for each pixel is done (realtime!) with the characteristics of the pixel range we look for (in this case bright pixels). SW1 is there to choose what to see on the TFT: normal camera images or the tresholded images. The image_treshold block also sends the VSYNC and HREF lines of the camera to the ili9325 block which is required to drive and synchronise the TFT to show the images.

So this was the basic description of all the blocks, those who want more details I invite to have a look in the VHDL codefiles for all these blocks.

It is now time to load the BIN file to your FPGA board (or use the .MCS file which is the FLASH memory config file, the difference is that when you use this then after power down and power up again the program is reloaded and when flashing the BIN file this is not the case). Loading the FPGA with its configuration file is done via iMpact from Xilinx. If this is not familiar to you then please have a look at the user manual of the FPGA board which I attached in Step 2.

I hope you enjoyed my Instructable and you also have a visual on your TFT screen like the example picture with this step shows! Here you see the result of very bright lights which are "detected" by the FPGA program: the treshold is set to very intense R,G and B levels in the pixels that the camera captures. The result is this black/white (tresholded) image.

I hope I will see lots of people intrigued by the power of an FPGA, picking up my FPGA vision and bring it to the next level!