DEMAC, a 3Dprinted Modular Beowulf Cluster

High Performance Computation (HPC) is is the ability to process data and perform complex calculations at high speeds, it is the application of "Supercomputers" to computational problems that are either too large for standard computers or would take too long to be completed. The Top500 is a list, that is published twice a year, and ranks some of the fastest, most powerful computers in the world. Countries and big organizations spend millions in resources to have this systems up and running for scientist to leverage state of the art technology and solve complex problems.

Years ago, computers used to improve their performance by increasing the speed of the processor. After we faced a slowdown in this kind of approach, developers decided that to continue increasing the performance of computers, multiple cores (or computational units) should be packed together. The aggregation of multiple computational resources and the mechanisms to control this resources is what we call "parallelism" in computer science. Having multiple cores doing multiple tasks sounds like a good approach to improve the performance of a computer ... but, this opens a big question: how do we use this resources more efficiently?

This questions have been keeping computer scientist busy, there are multiple ways to tell a computer how to do things, there are even more way to tell multiple computers how to do stuff. This project aims to develop an affordable platform where everyone can experiment with a highly parallel machine, test existing models to apply in your own projects, develop new and creative ways to solve computational problems or just use it as a way to teach others about computers. We hope you can enjoy working with DEMAC as much as we have.

DEMAC

The Delaware Modular Assembly Cluster (DEMAC) is an extendible array of embedded systems (card sized computers) and a set of 3Dprinted frames to encase the boards and additional hardware that provide power, cooling and network access.

Each device or embedded system is a small computer, a Parallella Board that combines the resources of a dual core ARM processor, a 16-core coprocessor called Epiphany and an embedded FPGA with the flexibility of a complete open-source stack. The mount is a house-made 3D-printed frame which allows for a low cost implementation and a escalable structure. It is designed to fit 4 units of a standard size rack (like those you find in computer servers rooms).

This instructable includes:

- A list of materials required

- Instructions to 3D print the frames

- Instructions to assemble and connect the parts

- A guide to download and install the required software

- A description of how to connect and interact with the cluster

- A "Why are we doing this?" section

Who are we?

We are CAPSL (Computer Architecture and Parallel Laboratory), from the University of Delaware. We believe the future of computation should have a strong base in Dataflow theory (which we will explain later in this instructable if you are interested).

This list describes the materials required to build a 4-board cluster

- 4 Parallella boards (you can get them from DigiKey or other vendors, you can find more information in their website https://www.parallella.org/)

- 4 micro-SD cards with at least 16Gb (here is a very cheap 10-pack or something like these more flexible combos)

- 4 micro-USB cables min length 30 cm (1 ft) (I recommend these)

- USB Charger [with at least 4 type A ports] (I recommend this one with 6-ports, or one with the same form factor, since the power case is designed for it)

- Cooling Fan [max size 100 mm x 100 mm x 15 mm] (I recommend this one because is cheap and works, but others with similar size and cable configuration works)

- Power supply for cooling fan (If your configuration is for more than 8 boards I recommend this one or something similar [AC 100 V/ 240 V to DC 12 V 10 A 120 W] that has a nice metal casing and can also be attached to the switch) (If you are going to plug just two fans or less you can use any 12 V with at least 1 A output power supply you may have lying around)

- 5 Ethernet Cables (4 can be short like these, depending on the distance from the switch to the boards, and one should be long enough to connect the switch with either your computer or the modem to access the cluster network)

>>> Important Note: A cooling system is required, otherwise the boards may overheat! <<<

3D-Printed Parts

- 4 Board Trays (Frame_01)

- 1 Board Casing (Frame_02)

- 1 Fan Casing (Frame_03_B & Frame_03_T)

- 1 Power Casing (Frame_04)

DEMAC is part of a bigger picture, a flexible and escalable platform that allows us to develop and test new programming execution models (PXM) for parallel computation. A PXM is more than a way to describe computation, it represents the backbone that provides an agreement between the way a program is expressed and how it is translated to a common language that can be executed by the machine. We describe a set of elements that allows the user to generate programs and a way to organize the execution of the program. The program can be optimized to target a specific architecture by the user or an automated tool based on this common background.

You can learn more about this project at the end of this instructable, you can also click here to get more information about DEMAC or here to get more information about CAPSL)

In this section you can find a guide to 3D print the frames that encase the other components and provide structural support. Even if you are a 3D printing master, here are some tips you can consider while printing these frames. All frames can be printed using a 0.4 mm nozzle with 0.3 or 0.2 layer height (you could also use adaptative). I printed everything using PLA but it doesn't really matter if you want to use other materials (as long as they provide structural stability and can tolerate higher or equal temperatures than PLA).

STL files:

https://www.thingiverse.com/thing:4493780

https://cults3d.com/en/3d-model/various/demac-a-mo...

https://www.myminifactory.com/object/3d-print-dema...

Board Tray (Frame_01)

No additional supports required. This one is pretty straightforward, just place it with the flat surface facing the printing surface.

Board Casing (Frame_02)

This one may require some support in the middle beams. You can argue that a well tuned machine/slicer can print those bridges without additional support. Please try some bridge stress tests first if you want to print without supports since the idea was that the wouldn't require them. On the other hand, the columns lateral and walls provide enough support for these to be printed without additional support structures.

Fan Casing (Frame_03_B & Frame_03_T)

No additional supports required. Just place both parts with the flat surface facing the printing surface.

Power Casing (Frame_04)

Similar to Frame_02, this one may require some support in the middle beams. You can also try to print this one without additional support material (as it was intended). The columns lateral and walls provide enough support for these to be printed without additional support structures.

Output Cooling Casing (Frame_05_B & Frame_05_T)

No additional supports required. Just place both parts with the flat surface facing the printing surface.

Now that you have all the parts required is time to start assembling the cluster.

Remember to remove the support material you might have on the frames.

Simply slide the fan inside Frame_03_B (with the cable in the right lower corner), the bottom part should fit inside the small curved walls that hold the fan in place.

Place Frame_03_T with the small curved walls facing downwards on top of Frame_03_B (with the fan already in place). Be careful to place the widest lid of Frame_03_T facing the wider (rear) face of Frame_03_B. The frames should click and the lids should keep them in place.

Place Frame_02 on top of Frame_04, these two are designed to snap together. There is a small dent on the bottom part of Frame_02 that fits the connectors on top of Frame_04. Apply gentle force to connect them.

Frame_03 (B&T) are designed to snap together with Frame_02, place the fan facing the boards (airflow should go inwards Frame_02). There are small dents on Frame_02's columns that should match the marks in Frame_03_B. Apply gentle pressure in the lateral faces of the structure until the frames click.

Frame_01 has 4 pins that match the holes in the Parallella board. The board should easily fit in the tray. Depending on your 3D-Printer calibration they might be to big or too small, you can use a bit of liquid silicone glue to hold them in place or press them a bit with some pliers to reduce the diameter.

>>> Important Note: Remember to place the heat-sinks on the board <<<

Frame_01 provide slots that fit into Frame_02 rails for each level. Note that there is only one side open to recieve the board tray. There is also a small bump that helps keeping Frame_01 in place (honestly, these could use some improvement in a future version).

Slide all 4 board trays with the boards already in place, 1 per each level.

Place the USB power supply inside Frame_04 with the USB ports facing outwards. There is a small opening on the other side for the power cable that feeds the hub.

The fan should now be connected to the 12 V power supply that provides energy to the cooling unit.

>>> Important Note: Keep the cooling system working all time while you have the boards connected to the power supply <<<

1. Download recommended OS (Parabuntu) here

There are two revisions of the chips (z7010 [P1600/P1601] and z7020 [P1602/A101040] which require different files.

For both revisions, there is a headless version (No graphical user intrface) and a version that provides HDMI support and a graphical user interface)

If you want to use the HDMI output remember to get a mini-HDMI cable.

You can interface with the headless version through the network.

More information and detailed explanation can be found here in the official website.

Here are the steps to install the operating system using a Linux-based distribution. You can use commands in the terminal (without the $ symbol) for the next steps or check other procedures in the website.

2. Install

- Insert the micro-SD card in your regular computer
- Unzip the Ubuntu image. Change [releasename] for the image name.

$ gunzip -d [releasename].img.gz

3. Verify the device path of your SD card

The exact device path to your SD card depends on your Linux distribution and computer setup. Using the command below to get the right path. If it’s not clear from output which path is the right one, try the command with and without the SD card inserted. In Ubuntu, the path returned might be something like ‘/dev/mmcblk0p1’.

$ df -h

4. Unmount the SD card
You will need to unmount all partitions on the SD cards before burning the card. The [sd-partition-path] comes from the ‘df’ command in step 3.

$ umount [sd-partition-path]

5. Burn the Ubuntu disk image on the micro-SD card

Burn the image onto the SD card using the ‘dd’ utility shown in the command example below. Please be careful and make sure you specify the path correctly as this command is irreversible and will overwrite anything in the path! An example command in Ubuntu would be: ‘sudo dd bs=4M if=my_release.img of=/dev/mmcblk0’. Please be patient, this could take a while (many minutes) depending on the computer and SD card being used.

$ sudo dd bs=4M if=[releasename].img of= [sd-partition-path]

6. Make sure all writes to the SD card have completed

$ sync

7. Insert SD card into the SD card slot in the board

Use the miniUSB to USB-A cable to connect one of the boards to the USB hub. You can label the ports and cables or define an order to the connections in case you need to disconnect a board later.

If you are doing the headless OS install while you’re on a large network, you will need to use a router and connect it to the internet, the Parallella boards, and your personal computer.

If you are not able to connect to the router, you can also connect the board directly to your computer using the Ethernet cable, this procedure can be a little trickier and would not be covered in this instructable.

Once everything is connected, open your router’s interface to find out what IP Address is being given to your Parallella by default. Look for a tab that says Network. Then find a section labeled DHCP Client List. There you should see your Parallella board and its IP Address.

With this IP Address, you can SSH into the Parallella and set up a static IP address.

Note: For this section, [default_IP] is the dynamic IP address you found in the DHCP Client List.

Check connection to the board

$ ping [default_IP]

SSH into the board for the first time (default password is parallella)

$ ssh parallella@[default_IP]

- Change hostname: edit /etc/hostname

Here you can assign any name you want, we recommend using NOPA##

Where ## identifies the board number (i.e. 01, 02, ...)

- Set other boards IP Addresses: edit /etc/hosts

Set static IP Address: add the below text for /etc/network/interfaces.d/eth0

#The primary network interface
auto eth0

iface eth0 inet static

address 192.168.10.101 #IP should be within router's range

netmask 255.255.255.0

gateway 192.168.10.1 #This should be the address of the router

nameserver 8.8.8.8

nameserver 8.8.4.4

Once you have assigned the IP to the board you can restart the connection with the command

$ ifdown eth0; ifup eth0

or reboot the board

Set up a private public key pair on every node (including the head node). Make a temporary folder, generate a new key and make it an authorized key, and add all NOPAs to the known hosts as shown below.

mkdir tmp_ssh
cd tmp_ssh ssh-keygen -f ./id_rsa

#Press enter twice to set and confirm an empty password

cp id_rsa.pub authorized_keys

for i in `seq 0 24`; do j=$(echo $i | awk '{printf "%02d\n", $0}');

ssh-keyscan NOPA$J >> known_hosts; done

- Using sshfs allow sharing files among the boards on the cluster. Run the following command:

$ sudo apt-get install -y sshfs

- Checking for / Creating Fuse Group

Check if the fuse group exists:

$ cat /etc/group | grep 'fuse'

If the group exists, execute the following command

$ bash sudo usermod -a -G fuse parallella

- If the group doesn’t exist, create it and add the user to it

$ sudo groupadd fuse

$ sudo usermod -a -G fuse parallella

- Uncomment the line user_allow_other in the file fuse.config

$ sudo vim /etc/fuse.conf

- Modify the file /etc/fstab

$ sudo vim /etc/fstab

- Replace the contents with the text shown below

# [file system] [mount point] [type] [options]

sshfs#parallella@NOPA01:/home/parallella/DEMAC_nfs /home/parallella/DEMAC_nfs fuse comment=sshfs,noauto,users,exec,rw,uid=1000,gid=1000,allow_other,reconnect,transform_symlinks,BatchMode=yes,nonempty,_netdev,identityfile=/home/parallella/.ssh/id_rsa,default_permissions 0 0

Place the switch below the cluster or somewhere nearby, use Ethernet cables to connect the board you have already configured to the switch. You can also connect the switch and your computer to the router to get access to the cluster.

You should be able to ping and ssh into the board that is now connected to the switch with a static IP.

You can also add the IP and hostname to your /etc/hosts file. You will be able to use the hostname to connect instead of typing the whole IP address.

Follow the procedure to configure the OS and network for each board.

>>> Important Note: Use different hostames and IP for each board! They should be unique through the network! <<<

Make sure the fan is working:

Make sure the fan is getting power and the airflow is going inwards into the Board casing. The connection should be stable and independent from other elements. Remember the boards can overheat if not cooled correctly.

Make sure the boards are connected to the switch:

At this point you should have configured each board independently. The boards should also be connected to the switch. The switch's manual should provide information that can be used to check that the startup process is completed correctly, there might be some LEDs that indicate status.

Connect the Boards to the Power Supply:

Use the micro-USB to USB-A cable to connect each of the boards to the USB hub. You can label the ports or define an order in case you need to disconnect a single board.

1. The Fan should be working.

2. The boards should be connected to the Ethernet switch.

3. Check that the boards are connected to the USB hub.

4. Provide power to the USB hub.

5. Enable DEMAC!

6. Profit!

MPI (Message Passing Interface)

MPI is a communication protocol for programming parallel computers. Both point-to-point and collective communication are supported.

https://www.open-mpi.org/

OpenMP (Open Multi-Processing)

The application programming interface (API) OpenMP (Open Multi-Processing) supports multi-platform shared-memory multiprocessing programming in C, C++, and Fortran, on many platforms. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior.

https://www.openmp.org/

Parallella Software

The developers provide an open-source software stack, including a SDK to interface with the accelerator.

http://www.parallella.org/software/

You can also find Manuals and more detailed information.

They also have GitHub repositories:

https://github.com/parallella

Feel free to download and run some of the examples, one of my favorites is game of life based on the famous Conway's Game of Life.

Disclaimer: Definitions might be copied from wikipedia